| jeblair | sdague, dtroyer: so i squashed them in https://review.openstack.org/#/c/112000 | 00:00 |
|---|---|---|
| clarkb | anteaya: its ok for stackforge to allow the release group to create branches IMO | 00:00 |
| clarkb | anteaya: so that is fine as is | 00:00 |
| anteaya | clarkb: ah okay so I'm wrong, thanks | 00:01 |
| *** tomoe_ has joined #openstack-infra | 00:01 | |
| fungi | anteaya: refs/heads/proposed/* would only let them create branches called proposed/something | 00:01 |
| clarkb | anteaya: the reason proposed is called out elsewhere is simply that openstack uses proposed/branches for things | 00:01 |
| anteaya | ah | 00:01 |
| anteaya | I'll update | 00:01 |
| jeblair | clarkb, fungi, sdague, dtroyer: i'm going to approve the squashed commit (since both were approved earlier, and this is gate-critical) | 00:02 |
| fungi | jeblair: wfm | 00:02 |
| jeblair | clarkb, fungi: and i'm going to promote it | 00:02 |
| clarkb | jeblair: sounds good | 00:02 |
| *** tomoe_ has quit IRC | 00:06 | |
| jeblair | done | 00:07 |
| *** dims has quit IRC | 00:07 | |
| *** sdake_ has quit IRC | 00:12 | |
| *** hashar has quit IRC | 00:12 | |
| *** wenlock has quit IRC | 00:14 | |
| *** ZZelle has quit IRC | 00:14 | |
| *** Ryan_Lane has quit IRC | 00:16 | |
| dtroyer | jeblair: for the record, cool. Also a mod 1000 review number | 00:17 |
| *** r-daneel has quit IRC | 00:17 | |
| *** amcrn has quit IRC | 00:19 | |
| fungi | more amazing is that we've now had 112 of those | 00:20 |
| openstackgerrit | K Jonathan Harker proposed a change to openstack-infra/config: Add single_use_slave coverage to the apply tests https://review.openstack.org/117389 | 00:20 |
| *** jamesmcarthur has joined #openstack-infra | 00:22 | |
| *** dmsimard_away is now known as dmsimard | 00:22 | |
| *** asettle has quit IRC | 00:22 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/config: Add list of types of logs to tempest-logs page https://review.openstack.org/117390 | 00:27 |
| jogo | fungi: thoughts ^ | 00:28 |
| *** hdd has quit IRC | 00:29 | |
| *** bhuvan has quit IRC | 00:30 | |
| fungi | jogo: first thought is that if we're doing the file indexes via the loganalyze front-end (e.g. with the swift backend) these descriptions could get integrated into the ui | 00:34 |
| clarkb | I have my glasses back in commission but a day was apparently long enough for my brain to get back into using the old ones | 00:34 |
| clarkb | jogo: fungi: ideally we would make the names better if they are hard to understand today | 00:35 |
| fungi | clarkb: ahh, yeah that too | 00:35 |
| clarkb | dtroyer might be able to speak to the choice for abbreviated log names | 00:35 |
| jogo | clarkb: yeah I think dropping the abbrevated names would be better | 00:36 |
| fungi | seconded | 00:36 |
| jogo | clarkb: not sure what the implications of that are | 00:36 |
| jogo | we should find out | 00:36 |
| clarkb | it will take more work (haev to update all logstash related stuff) | 00:36 |
| jogo | right | 00:36 |
| jogo | and a bunch of devstack changes | 00:36 |
| fungi | self-descriptive things are always the better route | 00:36 |
| jogo | but until then I am thinking this is better then nothing | 00:37 |
| *** mfer has joined #openstack-infra | 00:37 | |
| *** dane_leblanc has joined #openstack-infra | 00:37 | |
| jogo | assuming switching to better names takes more then just a few days | 00:37 |
| *** daneleblanc has joined #openstack-infra | 00:38 | |
| jogo | and then there are some logs that have 'good' names but are very short | 00:39 |
| *** baoli has quit IRC | 00:39 | |
| jogo | like pip-freeze | 00:39 |
| jogo | or iptables.txt | 00:39 |
| clarkb | jogo: is there something wrong with them? | 00:40 |
| clarkb | both of those are very descriptiev imo | 00:40 |
| *** yaguang has joined #openstack-infra | 00:41 | |
| jogo | clarkb: sure | 00:41 |
| fungi | i took that to be why he called them good names | 00:41 |
| jogo | clarkb: but you know what those things are | 00:41 |
| clarkb | fungi: the 'but' had me confused | 00:41 |
| clarkb | like being short was a bad thing | 00:41 |
| jogo | not everyone knows we use pip I bet | 00:42 |
| *** dane_leblanc has quit IRC | 00:42 | |
| *** dmsimard is now known as dmsimard_away | 00:42 | |
| fungi | those people are also not going to get far looking at our logs for other reasons | 00:42 |
| jogo | I think listing pip-freeze and dpkg-l under list of installed packages | 00:42 |
| jogo | would be useful | 00:42 |
| jogo | true | 00:42 |
| fungi | but sure, descriptions, even of those, can't hurt | 00:44 |
| *** chuckC has quit IRC | 00:45 | |
| *** nkinder has joined #openstack-infra | 00:46 | |
| *** baoli has joined #openstack-infra | 00:46 | |
| *** paulrad has joined #openstack-infra | 00:46 | |
| *** jamesmcarthur has quit IRC | 00:47 | |
| *** jamesmcarthur has joined #openstack-infra | 00:51 | |
| *** bdpayne has quit IRC | 00:51 | |
| dtroyer | clarkb: [catching up] are you referring to log names using the short service names? It seems like there are a couple of places where there is desire to use longer names. Honestly I did the short names way back to try and get them all in my screen status bar… | 00:53 |
| dtroyer | who knew we'd be a 47 and counting (I didn't count) | 00:54 |
| *** yamahata has joined #openstack-infra | 00:54 | |
| ianw | does anyone know off-hand how 'fqdn' gets set in -> https://github.com/puppetlabs/puppetlabs-mysql/blob/master/manifests/server/account_security.pp | 00:55 |
| jogo | dtroyer: yes, the short names get onfusing | 00:55 |
| ianw | i'm trying to verify bare-node setup on f20, it may be my problem in the vm setup | 00:55 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/config: Add list of types of logs to tempest-logs page https://review.openstack.org/117390 | 00:55 |
| jesusaurus | ianw: thats a fact being set by facter | 00:55 |
| jogo | dtroyer: when looking at http://logs.openstack.org/42/112942/4/check/check-tempest-dsvm-full/1ad11c8/logs/ | 00:55 |
| jogo | I had to look up what 'tr' stood for | 00:55 |
| jogo | dtroyer: any side effects of changine the names to long ones? | 00:56 |
| jogo | and dropping the 'screen' part | 00:56 |
| *** rick_ has joined #openstack-infra | 00:56 | |
| ianw | jesusaurus: what would cause it to not be set? puppet was dying because it seemed to be blank. | 00:56 |
| jogo | as its not from screen most of the time anyway | 00:56 |
| *** otherwiseguy has quit IRC | 00:56 | |
| *** yamahata has quit IRC | 00:57 | |
| dtroyer | jogo: re the log files, no. I was looking at that while workign on the process launching stuff today ad want to change it. it's backward compatability that concerns me, for everyone who isn't CI that has something automated… | 00:57 |
| *** yamahata has joined #openstack-infra | 00:57 | |
| *** Ryan_Lane has joined #openstack-infra | 00:57 | |
| jogo | dtroyer: yeah backwords compat ... | 00:58 |
| clarkb | dtroyer: ya the log names using short service names | 00:58 |
| jesusaurus | ianw: https://github.com/puppetlabs/facter/blob/master/lib/facter/fqdn.rb | 00:58 |
| jogo | dtroyer: can we opt in to the new naming sceme for now | 00:58 |
| jogo | scheme for now to support backwards compat | 00:58 |
| rick_ | anteaya: HI , I already disable to show "Starting check jobs." please check https://review.openstack.org/#/c/115886/ patch set 24. | 00:58 |
| jesusaurus | ianw: so is ::hostname also blank? | 00:58 |
| dtroyer | jogo: I think with a little thought we can do so. there are a couple of other logging things I'd like to clean up too | 00:58 |
| clarkb | ianw: you can test it on your machines with facter installed by doing `facter fqdn` | 00:58 |
| clarkb | ianw: jesusaurus: it won't be set if you don't have a domain longer than . | 00:59 |
| ianw | jesusaurus: hmm, ok. i'll have to check when puppet finishes. i guess the puppet run has wiped out my ssh key, so i'm locked out :) | 00:59 |
| clarkb | ianw: jesusaurus for whatever reason (and there are wontfix bugs on this) an fqdn according to facter must have a domain longer than . | 00:59 |
| jesusaurus | clarkb: oh right, trailing .s confuse it | 00:59 |
| clarkb | jesusaurus: well its not the trailing . | 00:59 |
| jogo | dtroyer: awesome | 01:00 |
| jogo | dtroyer: let me know if there is anything I can do to help with that | 01:00 |
| ianw | clarkb: ok, this is probably a cause of simply booting a rax vm and not setting anything | 01:00 |
| clarkb | its the lack of a domain. So while foo. resolves on my home dns as a fqdn facter says that is not an fqdn. You need foo.local. or simiarl | 01:00 |
| jogo | dtroyer: I was playing around with https://review.openstack.org/#/c/117390/ as a sort of workaround | 01:00 |
| jogo | but if the openstack file names become more descriptive that makes things esaier | 01:00 |
| anteaya | rick_: yes, I know your system is enabled again | 01:00 |
| rick_ | clarkb: How can I enable my service account? I already follow anteaya to hide "Starting check jobs." message. | 01:00 |
| anteaya | rick_: so work on the sandbox repo for a bit first | 01:01 |
| dtroyer | jogo: thanks, it's 'thinking' stuff first: these short names are also the names from ENABLED_SERVICES…switching to full names makes that longer and harder to scan for specific services in the logs…if we don't change those we set up mappings, which we may need anyway for a time. | 01:02 |
| anteaya | nibalizer: can I convince you to offer up a new patchset for https://review.openstack.org/#/c/96877/38 ? | 01:03 |
| rick_ | anteaya: does you mean I need trigger CI again? | 01:03 |
| dtroyer | I'm going to ponder this a bit…and try to not let it get too big | 01:03 |
| anteaya | nibalizer: Alex_Gaynor seems to be doing other things and I would really like livegrep | 01:03 |
| nibalizer | probably | 01:03 |
| jogo | dtroyer: awesome thanks. | 01:03 |
| anteaya | rick_: yes, your system is enabled, please limit yourself to testing the sandbox repo only for a few patches | 01:03 |
| nibalizer | i don t want to step on toes tho | 01:04 |
| anteaya | rick_: to ensure your system works well | 01:04 |
| anteaya | nibalizer: I understand, it has been a month with no activity | 01:04 |
| *** marcoemorais has quit IRC | 01:04 | |
| *** tomoe_ has joined #openstack-infra | 01:04 | |
| *** Ryan_Lane has quit IRC | 01:04 | |
| rick_ | anteaya: So I can change gerrit account to use "prophetstor-ci" | 01:04 |
| nibalizer | anteaya: okay yea ill add that to my list | 01:05 |
| anteaya | rick_: you can't change your gerrit account | 01:05 |
| nibalizer | right now im making the world safe for puppet 3 | 01:05 |
| anteaya | nibalizer: thanks | 01:05 |
| anteaya | nibalizer: good direction | 01:05 |
| nibalizer | but hopefully ill hit a stopping place soon | 01:05 |
| *** dims has joined #openstack-infra | 01:05 | |
| anteaya | nibalizer: one can only hope | 01:05 |
| anteaya | rick_: but you can use your gerrit account again | 01:05 |
| *** sdake_ has joined #openstack-infra | 01:06 | |
| rick_ | anteaya: OK | 01:06 |
| *** baoli has quit IRC | 01:06 | |
| anteaya | nibalizer: I just don't want to see livegrep get lost in the dust, I love it | 01:06 |
| *** hdd has joined #openstack-infra | 01:06 | |
| nibalizer | anteaya: okay i have an idea | 01:07 |
| *** jamesmcarthur has quit IRC | 01:08 | |
| rick_ | anteaya: I already trigger again. patch 25. https://review.openstack.org/#/c/115886/ | 01:09 |
| *** Ryan_Lane has joined #openstack-infra | 01:09 | |
| *** dims_ has joined #openstack-infra | 01:09 | |
| *** otherwiseguy has joined #openstack-infra | 01:10 | |
| *** dims__ has joined #openstack-infra | 01:11 | |
| *** dims has quit IRC | 01:11 | |
| *** blkperl has joined #openstack-infra | 01:12 | |
| anteaya | rick_: use your prophestor gerrit account to test the sandbox, not Rick Chen | 01:12 |
| rick_ | anteaya: But the prophetstor-ci account is disabled. Does it enable now? | 01:13 |
| nibalizer | blkperl: anteaya and i would like you to do a thing | 01:13 |
| blkperl | yes? | 01:13 |
| nibalizer | https://review.openstack.org/#/c/96877/38 | 01:13 |
| *** dkranz has joined #openstack-infra | 01:14 | |
| nibalizer | is kindof abandoned and we wants it | 01:14 |
| nibalizer | so if you could kick it over the wall that would be sweet | 01:14 |
| *** dims_ has quit IRC | 01:14 | |
| ianw | jesusaurus / clarkb : sorry for naive questions, still catching up with puppet. bare-node installation is failing on f20 due to postgres failures. i think https://github.com/puppetlabs/puppetlabs-postgresql/commit/0c4f2a96470cf0ce211757422b2db9f96e9ada07 fixes it. i guess there is no release with this; what are the options? | 01:14 |
| clarkb | ianw: you can try installing the postgres module via git. If you look in install_modules.sh there is an array of modules that we install via git now | 01:16 |
| clarkb | ianw: it takes a ref to install that module at | 01:16 |
| clarkb | however I would much prefer we don't do that | 01:16 |
| clarkb | maybe we can bribe krinkle to release a new version for us? | 01:17 |
| anteaya | nibalizer: everyone needs a blkperl | 01:17 |
| blkperl | lol | 01:17 |
| *** otherwiseguy has quit IRC | 01:17 | |
| *** terryw has joined #openstack-infra | 01:17 | |
| nibalizer | anteaya: i know right | 01:18 |
| anteaya | blkperl: while you're at it, the mob needs some screencasts, can you do that too? | 01:18 |
| * anteaya looks up editing software | 01:18 | |
| nibalizer | anteaya: what do we need screencasts of? | 01:18 |
| nibalizer | i actually really like doing those | 01:18 |
| anteaya | oh I hope sound libraries are fixed on trusty, I haven't tried yet | 01:18 |
| anteaya | cool | 01:18 |
| clarkb | fungi: jeblair: there is an awful lot of purple on our node graph | 01:18 |
| nibalizer | anteaya: don't count on it | 01:18 |
| ianw | clarkb: ok, testing with that ... | 01:19 |
| anteaya | how to submit an elastic recheck fingerprint should at least buy you some time | 01:19 |
| clarkb | looks like hpcloud may not be booting any nodes? | 01:19 |
| anteaya | I have a fantastic microphone I can use because of sound libraries | 01:19 |
| anteaya | :( | 01:19 |
| clarkb | https://community.hpcloud.com/status/incident/2724 that will do it | 01:19 |
| crinkle | clarkb: do you mean me? | 01:20 |
| anteaya | wow, awesome | 01:20 |
| anteaya | look at that dive | 01:20 |
| clarkb | crinkle: oh right you have adifferent name on this network | 01:20 |
| ianw | huh, puppet actually warns "Warning: Could not retrieve fact fqdn", but it's pretty easy to miss that | 01:20 |
| clarkb | crinkle: we dno't need to bribe you yet but maybe later | 01:20 |
| ianw | crinkle / clarkb : ok, top-of-tree puppetlabs-postgresql works and lets the bare-node install complete on f20 | 01:22 |
| nibalizer | crinkle: r-r-r-release! | 01:22 |
| *** mmaglana has quit IRC | 01:22 | |
| nibalizer | do et | 01:22 |
| clarkb | crinkle: we may need to bribe now. Not sure what your release process but aiui we can't use postgres module on fedora right now | 01:23 |
| crinkle | our release process is a little bloated and postgres isn't scheduled to be released this week | 01:23 |
| nibalizer | crinkle: are you saying its scheduled to be broken this week? | 01:23 |
| crinkle | i could try to ask for it to happen next week | 01:23 |
| nibalizer | crinkle: i meant that as a joke, sorry if it came out wrong | 01:24 |
| crinkle | :P | 01:24 |
| rick_ | anteaya: Thanks. All is fine. | 01:24 |
| *** mmedvede has joined #openstack-infra | 01:25 | |
| ianw | clarkb / crinkle : i'll propose the install_modules work-around. i'll keep on it and remove it when a suitable release comes | 01:26 |
| clarkb | ianw: sounds good | 01:26 |
| *** jpeeler has quit IRC | 01:26 | |
| anteaya | rick_: good work | 01:27 |
| openstackgerrit | A change was merged to openstack-infra/config: Fix DEVSTACK_GATE_GRENADE values for sideways jobs https://review.openstack.org/117333 | 01:28 |
| rick_ | anteaya: many thanks for you. | 01:28 |
| anteaya | nibalizer: well audacity on trusty can find my external edirol soundcard and my neumann microphone, so that is a start | 01:28 |
| anteaya | rick_: np, go forward and help others | 01:28 |
| anteaya | rick_: and attend third party meetings when you are able | 01:29 |
| rick_ | anteaya: OK, | 01:29 |
| nibalizer | anteaya: thats a good sign | 01:31 |
| nibalizer | i think it was clarkb that had to rollback his trusty upgrade because ENOSOUND | 01:31 |
| *** adalbas has joined #openstack-infra | 01:31 | |
| *** sdake_ has quit IRC | 01:31 | |
| anteaya | nibalizer: I had to do that with either q or r | 01:32 |
| *** gyee has quit IRC | 01:32 | |
| *** asettle has joined #openstack-infra | 01:32 | |
| clarkb | nibalizer: I didn't rollback. I did try turning on built in sound thing with no luck | 01:33 |
| nibalizer | clarkb: oh are you just soundless? | 01:33 |
| clarkb | which makes me think it isn't a driver issue but something futher up the stack /me glares at pulseaudio | 01:33 |
| nibalizer | my condolences | 01:33 |
| clarkb | nibalizer: yup | 01:33 |
| *** trinaths has quit IRC | 01:35 | |
| *** baoli has joined #openstack-infra | 01:40 | |
| nibalizer | according to puppetboard looks like our master hasn't run in about a month | 01:42 |
| *** mriedem has joined #openstack-infra | 01:45 | |
| clarkb | nibalizer: I don't know what that means | 01:45 |
| clarkb | you mean puppet hasn't run on the master in about a month? | 01:46 |
| *** mriedem has quit IRC | 01:46 | |
| *** sdake_ has joined #openstack-infra | 01:46 | |
| clarkb | nibalizer: it looks fine to me | 01:47 |
| *** mmedvede has quit IRC | 01:47 | |
| openstackgerrit | A change was merged to openstack-infra/config: Import pylockfile https://review.openstack.org/101911 | 01:48 |
| *** adalbas has quit IRC | 01:49 | |
| *** nosnos has joined #openstack-infra | 01:49 | |
| openstackgerrit | Ian Wienand proposed a change to openstack-infra/config: Get postgresql puppet from upstream master https://review.openstack.org/117396 | 01:50 |
| openstackgerrit | Ian Wienand proposed a change to openstack-infra/config: Add bare-f20 nodes https://review.openstack.org/117397 | 01:50 |
| *** homeless has quit IRC | 01:51 | |
| *** r1chardj0n3s is now known as r1chardj0n3s_afk | 01:56 | |
| *** baohua has joined #openstack-infra | 01:57 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: clean up readme https://review.openstack.org/115011 | 01:59 |
| *** klindgren__ has quit IRC | 01:59 | |
| *** esker has joined #openstack-infra | 02:00 | |
| *** Ryan_Lane has quit IRC | 02:01 | |
| *** openstackgerrit has quit IRC | 02:02 | |
| *** openstackgerrit has joined #openstack-infra | 02:02 | |
| *** hogepodge has joined #openstack-infra | 02:03 | |
| *** dkranz has quit IRC | 02:06 | |
| greghaynes | pleia2: Are you in SFO? | 02:08 |
| * greghaynes might have spied a pleia2 | 02:08 | |
| clarkb | greghaynes: she is flying to pdx today sometime for debconf | 02:08 |
| clarkb | greghaynes: so maybe even on your flight | 02:08 |
| greghaynes | wow | 02:08 |
| greghaynes | hogepodge: and I are here | 02:08 |
| pleia2 | i am! where are you? | 02:10 |
| *** sflanigan__ is now known as sflanigan | 02:10 | |
| pleia2 | my flight is delayed, boo | 02:10 |
| clarkb | pleia2: how long will you be in pdx for the conference? | 02:10 |
| greghaynes | pleia2: were in the food court | 02:10 |
| greghaynes | pleia2: come over, were sitting next to the pilot of the plane were all on | 02:10 |
| greghaynes | sounds like were on the same flight | 02:10 |
| pleia2 | woo | 02:10 |
| *** jamesmcarthur has joined #openstack-infra | 02:11 | |
| pleia2 | clarkb: leaving sunday | 02:11 |
| *** sflanigan has quit IRC | 02:11 | |
| greghaynes | pilot is like "yeaaa this sucks" | 02:11 |
| *** sflanigan has joined #openstack-infra | 02:11 | |
| *** sflanigan has joined #openstack-infra | 02:11 | |
| pleia2 | haha | 02:11 |
| clarkb | united? | 02:13 |
| greghaynes | yep | 02:13 |
| clarkb | I think I found your problem | 02:13 |
| *** dims__ has quit IRC | 02:13 | |
| *** mfer has quit IRC | 02:14 | |
| clarkb | pleia2: hrm ok can you drink yet? I am trying to device homebrew imbibing but was thinking of possibling driving down sunday | 02:14 |
| *** dims has joined #openstack-infra | 02:14 | |
| greghaynes | doit | 02:14 |
| greghaynes | 1 | 02:14 |
| * greghaynes is in | 02:14 | |
| pleia2 | clarkb: I can drink! | 02:18 |
| clarkb | might have to drive down saturday afternoon then | 02:20 |
| openstackgerrit | A change was merged to openstack/requirements: Add oslo.serialization https://review.openstack.org/117239 | 02:20 |
| anteaya | so far audacity/vlc for capture/openshot for editing all seem to be working | 02:20 |
| anteaya | tell me what format/codecs I should be using before I use the wrong thing | 02:21 |
| * anteaya is completely surprised she found tooling this fast | 02:21 | |
| anteaya | I never could before | 02:21 |
| clarkb | mp4? | 02:22 |
| clarkb | I actually don't know | 02:22 |
| *** jpeeler has joined #openstack-infra | 02:23 | |
| anteaya | well mp4 is what I have as output, with mp4 for screencapture and .wav for audio as input | 02:24 |
| anteaya | but I don't know if that is what I should be doing or not | 02:24 |
| anteaya | those are just the defaults on the apps | 02:25 |
| *** jamesmcarthur has quit IRC | 02:26 | |
| *** _nadya_ has joined #openstack-infra | 02:32 | |
| *** asettle has quit IRC | 02:33 | |
| *** asettle has joined #openstack-infra | 02:37 | |
| *** dkehn has joined #openstack-infra | 02:39 | |
| *** chuckC has joined #openstack-infra | 02:40 | |
| *** dkehn_ has quit IRC | 02:40 | |
| *** dkehnx has quit IRC | 02:40 | |
| *** jamesmcarthur has joined #openstack-infra | 02:41 | |
| *** dkehn_ has joined #openstack-infra | 02:42 | |
| *** stevemar has joined #openstack-infra | 02:45 | |
| *** _nadya_ has quit IRC | 02:45 | |
| *** KanagarajM has joined #openstack-infra | 02:47 | |
| *** paulrad has quit IRC | 02:50 | |
| *** melwitt has quit IRC | 02:56 | |
| *** asettle has quit IRC | 03:04 | |
| *** _nadya_ has joined #openstack-infra | 03:04 | |
| *** sdake_ has quit IRC | 03:05 | |
| *** sdake_ has joined #openstack-infra | 03:06 | |
| *** hogepodge has quit IRC | 03:06 | |
| jesusaurus | anteaya: i would suggest mp4 with h264/x264 video and ac3 or aac audio. that should give you a good balance of quality and portability across platforms | 03:09 |
| *** rick_ has quit IRC | 03:10 | |
| *** otter768 has joined #openstack-infra | 03:15 | |
| *** hogepodge has joined #openstack-infra | 03:17 | |
| *** r1chardj0n3s_afk is now known as r1chardj0n3s | 03:18 | |
| nibalizer | anteaya: when i do screencasting i do it straight to twitch | 03:20 |
| nibalizer | there is an ffmpeg line to do it, its magic | 03:20 |
| *** mmaglana has joined #openstack-infra | 03:24 | |
| *** nosnos has quit IRC | 03:26 | |
| *** jamesmcarthur has left #openstack-infra | 03:27 | |
| *** pballand has joined #openstack-infra | 03:27 | |
| *** nosnos has joined #openstack-infra | 03:27 | |
| *** _nadya_ has quit IRC | 03:27 | |
| *** hogepodge has quit IRC | 03:31 | |
| *** Longgeek has joined #openstack-infra | 03:31 | |
| *** nosnos has quit IRC | 03:32 | |
| *** daya_k has joined #openstack-infra | 03:36 | |
| *** praneshp_ has quit IRC | 03:41 | |
| *** otter768 has quit IRC | 03:51 | |
| *** hanzhang has quit IRC | 03:58 | |
| *** pballand has quit IRC | 04:00 | |
| *** praneshp has joined #openstack-infra | 04:03 | |
| *** daneleblanc has quit IRC | 04:03 | |
| *** pballand has joined #openstack-infra | 04:05 | |
| *** pballand has quit IRC | 04:09 | |
| *** garyh has quit IRC | 04:09 | |
| *** asettle has joined #openstack-infra | 04:14 | |
| *** dims has quit IRC | 04:17 | |
| *** dane_leblanc has joined #openstack-infra | 04:19 | |
| *** pcrews has quit IRC | 04:19 | |
| *** bhuvan has joined #openstack-infra | 04:20 | |
| *** lukego has joined #openstack-infra | 04:22 | |
| *** nosnos has joined #openstack-infra | 04:27 | |
| *** trinaths has joined #openstack-infra | 04:34 | |
| *** amotoki has joined #openstack-infra | 04:36 | |
| *** bhuvan has quit IRC | 04:38 | |
| *** emagana has joined #openstack-infra | 04:40 | |
| *** yfried has quit IRC | 04:48 | |
| *** stevemar has quit IRC | 04:48 | |
| *** Longgeek_ has joined #openstack-infra | 04:52 | |
| *** emagana has quit IRC | 04:55 | |
| *** Longgeek has quit IRC | 04:56 | |
| *** koolhead17 has joined #openstack-infra | 04:59 | |
| SergeyLukjanov | jeblair, thank you for pushing 112000 | 05:00 |
| *** garyh has joined #openstack-infra | 05:01 | |
| *** nelsnelson has quit IRC | 05:02 | |
| *** nelsnelson has joined #openstack-infra | 05:06 | |
| *** asettle has quit IRC | 05:08 | |
| *** koolhead17 has quit IRC | 05:11 | |
| trinaths | Is jenkins okay, I see its taking much time | 05:16 |
| *** terryw has quit IRC | 05:17 | |
| *** dims has joined #openstack-infra | 05:18 | |
| *** hdd has quit IRC | 05:19 | |
| *** dims has quit IRC | 05:22 | |
| openstackgerrit | A change was merged to openstack/requirements: Bump futures to 2.1.6 for swiftclient https://review.openstack.org/117216 | 05:29 |
| openstackgerrit | A change was merged to openstack/requirements: Revert "Remove docutils pin" https://review.openstack.org/117168 | 05:29 |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Add Ironic-python-agent support to devstack-gate https://review.openstack.org/112143 | 05:39 |
| *** reed has quit IRC | 05:41 | |
| *** oomichi has quit IRC | 05:42 | |
| *** loki184 has joined #openstack-infra | 05:44 | |
| *** yfried has joined #openstack-infra | 05:44 | |
| *** ppai has joined #openstack-infra | 05:50 | |
| *** _nadya_ has joined #openstack-infra | 05:53 | |
| *** emagana has joined #openstack-infra | 05:56 | |
| *** leifmadsen has quit IRC | 05:57 | |
| *** emagana has quit IRC | 06:00 | |
| *** _nadya_ has quit IRC | 06:01 | |
| *** leifmadsen has joined #openstack-infra | 06:02 | |
| *** k4n0 has joined #openstack-infra | 06:03 | |
| *** ociuhandu has quit IRC | 06:04 | |
| *** ociuhandu has joined #openstack-infra | 06:05 | |
| *** _buzz has quit IRC | 06:06 | |
| *** emagana has joined #openstack-infra | 06:12 | |
| *** emagana has quit IRC | 06:17 | |
| *** HeOS has quit IRC | 06:24 | |
| *** flaper87|afk is now known as flaper87 | 06:27 | |
| *** trinaths_afk has joined #openstack-infra | 06:27 | |
| *** trinaths has quit IRC | 06:28 | |
| *** trinaths_afk is now known as trinaths | 06:28 | |
| *** dtantsur|afk is now known as dtantsur | 06:30 | |
| *** ildikov has quit IRC | 06:32 | |
| *** e0ne has joined #openstack-infra | 06:37 | |
| *** Ryan_Lane has joined #openstack-infra | 06:40 | |
| *** jlibosva has joined #openstack-infra | 06:41 | |
| openstackgerrit | Adam Gandelman proposed a change to openstack-infra/devstack-gate: Fix sideways grenade local tmp variables https://review.openstack.org/117435 | 06:41 |
| *** afazekas has joined #openstack-infra | 06:41 | |
| *** e0ne has quit IRC | 06:43 | |
| *** flaviof is now known as flaviof_zzz | 06:54 | |
| *** denis_makogon has joined #openstack-infra | 06:54 | |
| *** ildikov has joined #openstack-infra | 06:54 | |
| *** mpaolino has joined #openstack-infra | 06:55 | |
| *** HeOS has joined #openstack-infra | 06:56 | |
| *** e0ne has joined #openstack-infra | 06:57 | |
| *** e0ne has quit IRC | 06:58 | |
| lyxus | Is there an issue with zuul ? Remaining time for a job that I am monitoring is went from 1h to 1h17min to unknown | 07:00 |
| *** dtantsur is now known as dtantsur|bbl | 07:03 | |
| *** emagana has joined #openstack-infra | 07:13 | |
| *** marun has joined #openstack-infra | 07:14 | |
| *** harlowja_ is now known as harlowja_away | 07:14 | |
| *** _nadya_ has joined #openstack-infra | 07:15 | |
| *** emagana has quit IRC | 07:17 | |
| trinaths | lyxus: cool, its taking up now.. hope things are getting better now | 07:18 |
| lyxus | trinaths, It is, just wish this review will be merge faster:) | 07:19 |
| lyxus | *merged | 07:19 |
| *** esker has quit IRC | 07:21 | |
| *** HeOS has quit IRC | 07:22 | |
| *** Hal_ has joined #openstack-infra | 07:23 | |
| *** todd_dsm has quit IRC | 07:24 | |
| *** dizquierdo has joined #openstack-infra | 07:27 | |
| *** jgallard has joined #openstack-infra | 07:28 | |
| trinaths | lyxus: :) | 07:28 |
| *** HeOS has joined #openstack-infra | 07:29 | |
| *** jgallard has quit IRC | 07:35 | |
| *** jgallard has joined #openstack-infra | 07:35 | |
| marun | Does anyone know if the pg job failures are temporal or need a fix? | 07:36 |
| *** mflobo_ has joined #openstack-infra | 07:37 | |
| *** andreykurilin has joined #openstack-infra | 07:39 | |
| *** mflobo has quit IRC | 07:41 | |
| *** cristov_mac has joined #openstack-infra | 07:45 | |
| *** cristov_mac has quit IRC | 07:45 | |
| *** jcoufal has joined #openstack-infra | 07:49 | |
| *** rpodolyaka has joined #openstack-infra | 07:52 | |
| *** jpich has joined #openstack-infra | 07:53 | |
| *** IvanBerezovskiy has joined #openstack-infra | 07:55 | |
| *** jistr has joined #openstack-infra | 07:56 | |
| *** jistr has quit IRC | 07:56 | |
| *** mpaolino has quit IRC | 07:58 | |
| *** cnesa8 has joined #openstack-infra | 08:00 | |
| *** trinaths has quit IRC | 08:01 | |
| openstackgerrit | A change was merged to openstack-infra/config: Correct merge calculation error in layout.yaml https://review.openstack.org/117375 | 08:01 |
| *** mancdaz has quit IRC | 08:02 | |
| *** jgallard has quit IRC | 08:06 | |
| *** jistr has joined #openstack-infra | 08:08 | |
| *** tomoe_ has quit IRC | 08:08 | |
| *** r1chardj0n3s is now known as r1chardj0n3s_afk | 08:08 | |
| *** jgallard has joined #openstack-infra | 08:09 | |
| *** tomoe_ has joined #openstack-infra | 08:10 | |
| *** jgallard has quit IRC | 08:11 | |
| *** trinaths has joined #openstack-infra | 08:11 | |
| *** jgallard has joined #openstack-infra | 08:11 | |
| *** emagana has joined #openstack-infra | 08:14 | |
| *** praneshp has quit IRC | 08:17 | |
| *** emagana has quit IRC | 08:19 | |
| *** e0ne has joined #openstack-infra | 08:19 | |
| *** derekh has joined #openstack-infra | 08:20 | |
| *** flaper87 is now known as flaper87|afk | 08:21 | |
| *** boris-42 has quit IRC | 08:21 | |
| *** pblaho has joined #openstack-infra | 08:23 | |
| *** pblaho has joined #openstack-infra | 08:24 | |
| *** skolekonov has joined #openstack-infra | 08:25 | |
| *** doude has joined #openstack-infra | 08:25 | |
| *** boris-42 has joined #openstack-infra | 08:30 | |
| *** flaper87|afk is now known as flaper87 | 08:30 | |
| *** jp_at_hp has joined #openstack-infra | 08:32 | |
| *** hashar has joined #openstack-infra | 08:33 | |
| *** jamielennox is now known as jamielennox|away | 08:36 | |
| *** zz_johnthetubagu is now known as johnthetubaguy | 08:43 | |
| *** tomoe__ has joined #openstack-infra | 08:44 | |
| *** tomoe___ has joined #openstack-infra | 08:46 | |
| *** tomoe_ has quit IRC | 08:46 | |
| *** andreykurilin has quit IRC | 08:47 | |
| *** nosnos has quit IRC | 08:47 | |
| *** nosnos has joined #openstack-infra | 08:47 | |
| *** tomoe__ has quit IRC | 08:48 | |
| *** tomoe_ has joined #openstack-infra | 08:49 | |
| *** mflobo_ has quit IRC | 08:50 | |
| *** mflobo has joined #openstack-infra | 08:50 | |
| *** tomoe___ has quit IRC | 08:51 | |
| *** nosnos has quit IRC | 08:52 | |
| *** tomoe__ has joined #openstack-infra | 08:52 | |
| *** tomoe_ has quit IRC | 08:54 | |
| *** joools has joined #openstack-infra | 08:55 | |
| *** tomoe_ has joined #openstack-infra | 08:57 | |
| *** tomoe__ has quit IRC | 08:58 | |
| *** hashar_ has joined #openstack-infra | 08:59 | |
| *** jgallard has quit IRC | 09:00 | |
| *** markmc has joined #openstack-infra | 09:01 | |
| *** joools has quit IRC | 09:01 | |
| *** hashar has quit IRC | 09:02 | |
| *** hashar_ is now known as hashar | 09:05 | |
| *** yongli has quit IRC | 09:06 | |
| *** rlandy has joined #openstack-infra | 09:09 | |
| *** ociuhandu has quit IRC | 09:15 | |
| *** emagana has joined #openstack-infra | 09:16 | |
| *** alexpilotti has joined #openstack-infra | 09:18 | |
| openstackgerrit | Romain Chantereau proposed a change to openstack-infra/config: Be able to generate math formulae with Sphinx mathpng base extension. https://review.openstack.org/117263 | 09:20 |
| *** emagana has quit IRC | 09:21 | |
| openstackgerrit | Romain Chantereau proposed a change to openstack-infra/config: Be able to generate math formulae with Sphinx mathpng base extension. https://review.openstack.org/117263 | 09:22 |
| *** AlecTaylor has joined #openstack-infra | 09:23 | |
| *** nosnos has joined #openstack-infra | 09:24 | |
| lyxus | trinaths, are you still here ? | 09:34 |
| trinaths | lyxus, yes | 09:39 |
| lyxus | trinaths, I waiting for 114402,10 to be merged. It has 3 min left with all of the tests done but one. And I just checked and it has 58 minutes left and re-running all of the tests. Would you know what is happening ? | 09:41 |
| *** luqas has joined #openstack-infra | 09:42 | |
| trinaths | lyxus: nope. You mean GATE pipeline ? I'm not sure of i | 09:43 |
| trinaths | it | 09:43 |
| lyxus | trinaths, yup GATE | 09:43 |
| trinaths | lyxus, donno | 09:44 |
| lyxus | trinaths, ok thank you ! | 09:44 |
| *** amuller has joined #openstack-infra | 09:44 | |
| *** yamahata has quit IRC | 09:45 | |
| *** linkid has joined #openstack-infra | 09:47 | |
| *** oomichi has joined #openstack-infra | 09:47 | |
| openstackgerrit | Dmitry Shulyak proposed a change to openstack-infra/config: Create fuel-tasklib repository https://review.openstack.org/117466 | 09:51 |
| *** lukego has quit IRC | 09:52 | |
| *** mpaolino has joined #openstack-infra | 09:54 | |
| *** Hal_ has quit IRC | 09:54 | |
| *** unicell has quit IRC | 09:58 | |
| *** michchap has quit IRC | 09:58 | |
| *** isviridov is now known as isviridov_awa | 10:00 | |
| *** isviridov_awa is now known as isviridov_awat | 10:00 | |
| *** isviridov_awat is now known as isviridov_away | 10:00 | |
| *** baohua has quit IRC | 10:01 | |
| *** michchap has joined #openstack-infra | 10:03 | |
| *** jgallard has joined #openstack-infra | 10:05 | |
| *** e0ne has quit IRC | 10:05 | |
| *** mpaolino has quit IRC | 10:06 | |
| *** jcoufal has quit IRC | 10:09 | |
| *** jab has joined #openstack-infra | 10:10 | |
| *** jab has joined #openstack-infra | 10:10 | |
| *** jcoufal has joined #openstack-infra | 10:12 | |
| openstackgerrit | Salvatore Orlando proposed a change to openstack-infra/elastic-recheck: Add footprint for bug 1362538 https://review.openstack.org/117468 | 10:12 |
| *** denis_makogon has quit IRC | 10:13 | |
| *** e0ne has joined #openstack-infra | 10:13 | |
| *** jgallard has quit IRC | 10:15 | |
| *** yolanda_ has joined #openstack-infra | 10:15 | |
| *** kmartin has quit IRC | 10:15 | |
| *** yolanda has quit IRC | 10:16 | |
| *** emagana has joined #openstack-infra | 10:17 | |
| *** jab is now known as bradjones | 10:20 | |
| *** nosnos has quit IRC | 10:20 | |
| *** nosnos has joined #openstack-infra | 10:21 | |
| *** emagana has quit IRC | 10:21 | |
| *** nosnos has quit IRC | 10:25 | |
| *** pblaho is now known as pblaho|afk | 10:29 | |
| *** pelix1 has joined #openstack-infra | 10:32 | |
| *** pblaho|afk is now known as pblaho | 10:32 | |
| *** koolhead17 has joined #openstack-infra | 10:33 | |
| *** koolhead17 has quit IRC | 10:34 | |
| *** nosnos has joined #openstack-infra | 10:34 | |
| *** lukego has joined #openstack-infra | 10:36 | |
| *** isviridov_away is now known as isviridov | 10:37 | |
| *** ppai has quit IRC | 10:38 | |
| *** luqas has quit IRC | 10:38 | |
| *** tomoe_ has quit IRC | 10:42 | |
| *** HenryG has quit IRC | 10:43 | |
| openstackgerrit | Dirk Mueller proposed a change to openstack-dev/pbr: use get_boolean_option for warnerrors https://review.openstack.org/117476 | 10:43 |
| *** unicell has joined #openstack-infra | 10:44 | |
| openstackgerrit | Dirk Mueller proposed a change to openstack-dev/pbr: Allow overwriting sphinx builder list from command line https://review.openstack.org/117478 | 10:44 |
| rakhmerov | hello, jenkins job queue seems to be consistenly growing. Is something wrong with it or it's just a huge load? | 10:46 |
| openstackgerrit | Dirk Mueller proposed a change to openstack-dev/pbr: Allow overwriting sphinx builder list from command line https://review.openstack.org/117478 | 10:47 |
| *** ppai has joined #openstack-infra | 10:52 | |
| *** AlecTaylor has quit IRC | 10:53 | |
| *** Hal_ has joined #openstack-infra | 10:58 | |
| *** KanagarajM has quit IRC | 11:01 | |
| *** yaguang has quit IRC | 11:01 | |
| *** ociuhandu has joined #openstack-infra | 11:02 | |
| *** pblaho is now known as pblaho|afk | 11:04 | |
| *** ildikov has quit IRC | 11:05 | |
| openstackgerrit | A change was merged to openstack-dev/pbr: Handle more legacy version numbers https://review.openstack.org/116775 | 11:06 |
| *** ildikov has joined #openstack-infra | 11:07 | |
| *** bradjones has quit IRC | 11:08 | |
| *** loki184 has quit IRC | 11:08 | |
| isviridov | SergeyLukjanov, fungi the zuul queue is really huge with patches submitted 11 hours ago | 11:09 |
| *** ociuhandu has quit IRC | 11:11 | |
| *** ildikov has quit IRC | 11:12 | |
| *** emagana has joined #openstack-infra | 11:18 | |
| *** nosnos has quit IRC | 11:18 | |
| *** nosnos has joined #openstack-infra | 11:20 | |
| *** emagana has quit IRC | 11:22 | |
| *** flaviof_zzz is now known as flaviof | 11:23 | |
| *** mattymo has quit IRC | 11:28 | |
| *** daya_k has quit IRC | 11:30 | |
| openstackgerrit | yolanda.robla proposed a change to openstack-infra/publications: New CI Automation doc https://review.openstack.org/109570 | 11:32 |
| *** trinaths_afk has joined #openstack-infra | 11:33 | |
| *** mattymo has joined #openstack-infra | 11:35 | |
| *** trinaths has quit IRC | 11:35 | |
| *** trinaths_afk is now known as trinaths | 11:35 | |
| *** linkid has left #openstack-infra | 11:37 | |
| *** linkid has joined #openstack-infra | 11:37 | |
| *** marcusvrn has joined #openstack-infra | 11:42 | |
| *** dane_leblanc has quit IRC | 11:42 | |
| *** jgallard has joined #openstack-infra | 11:44 | |
| *** dtantsur|bbl is now known as dtantsur | 11:44 | |
| *** pcm_ has joined #openstack-infra | 11:49 | |
| openstackgerrit | Cedric Brandily proposed a change to openstack-infra/git-review: -F/--force-rebase has no effect if rebase is disabled by config https://review.openstack.org/117489 | 11:50 |
| fungi | yes, according to the scrollback hpcloud was basically down for us for a while, and may still be. i'll look for updates once i'm awake and have coffee | 11:51 |
| *** HenryG has joined #openstack-infra | 11:52 | |
| fungi | feature proposal procrastination week is always bad though | 11:52 |
| sdague | it also times badly with holidays now. As we also get a push with people coming back from aug vacations and still trying to land things | 11:55 |
| rakhmerov | ok | 11:55 |
| sdague | IIRC for folsom the freeze was actually aug 15th | 11:55 |
| *** dane_leblanc has joined #openstack-infra | 11:57 | |
| *** hashar has quit IRC | 11:58 | |
| *** _nadya_ has quit IRC | 12:02 | |
| *** _nadya_ has joined #openstack-infra | 12:02 | |
| *** pcm_ has quit IRC | 12:03 | |
| *** e0ne has quit IRC | 12:04 | |
| sdague | fungi: where are the all-projects acl set in config? | 12:05 |
| *** kgiusti has joined #openstack-infra | 12:05 | |
| *** jab has joined #openstack-infra | 12:05 | |
| *** jab has joined #openstack-infra | 12:05 | |
| *** _buzz has joined #openstack-infra | 12:06 | |
| *** yfried has quit IRC | 12:06 | |
| *** afazekas has quit IRC | 12:06 | |
| *** jab is now known as bradjones | 12:07 | |
| *** mmaglana has quit IRC | 12:07 | |
| *** yfried has joined #openstack-infra | 12:07 | |
| *** yfried has quit IRC | 12:08 | |
| *** yfried has joined #openstack-infra | 12:08 | |
| *** yfried_ has joined #openstack-infra | 12:10 | |
| *** masayukig has quit IRC | 12:10 | |
| *** dprince has joined #openstack-infra | 12:11 | |
| openstackgerrit | Ian Wienand proposed a change to openstack-infra/config: Get postgresql puppet from upstream master https://review.openstack.org/117396 | 12:11 |
| *** mmedvede has joined #openstack-infra | 12:11 | |
| *** pblaho|afk is now known as pblaho | 12:11 | |
| *** yfried has quit IRC | 12:13 | |
| *** pcm_ has joined #openstack-infra | 12:13 | |
| *** masayukig has joined #openstack-infra | 12:15 | |
| *** weshay has joined #openstack-infra | 12:16 | |
| *** e0ne has joined #openstack-infra | 12:18 | |
| *** emagana has joined #openstack-infra | 12:18 | |
| *** ominakov has joined #openstack-infra | 12:20 | |
| *** adalbas has joined #openstack-infra | 12:21 | |
| *** nosnos has quit IRC | 12:22 | |
| *** nosnos has joined #openstack-infra | 12:23 | |
| *** emagana has quit IRC | 12:23 | |
| *** aysyd has joined #openstack-infra | 12:24 | |
| *** lukego has quit IRC | 12:24 | |
| *** trinaths has quit IRC | 12:25 | |
| *** hashar has joined #openstack-infra | 12:25 | |
| *** _nadya_ has quit IRC | 12:26 | |
| *** nosnos has quit IRC | 12:27 | |
| *** sdake_ has quit IRC | 12:28 | |
| *** acfleury has joined #openstack-infra | 12:28 | |
| *** yamahata has joined #openstack-infra | 12:30 | |
| *** julim has joined #openstack-infra | 12:33 | |
| *** sdake_ has joined #openstack-infra | 12:35 | |
| *** julim has quit IRC | 12:35 | |
| *** acfleury has quit IRC | 12:36 | |
| *** mmaglana has joined #openstack-infra | 12:38 | |
| *** julim has joined #openstack-infra | 12:39 | |
| *** mmaglana_ has joined #openstack-infra | 12:40 | |
| *** lukego has joined #openstack-infra | 12:40 | |
| *** dizquierdo is now known as dizquierdo_afk | 12:41 | |
| *** mmaglan__ has joined #openstack-infra | 12:42 | |
| *** hdd has joined #openstack-infra | 12:42 | |
| *** ominakov has quit IRC | 12:42 | |
| *** mmaglana has quit IRC | 12:43 | |
| *** ppai has quit IRC | 12:44 | |
| *** mmaglana_ has quit IRC | 12:44 | |
| *** marun has quit IRC | 12:45 | |
| *** mmaglan__ has quit IRC | 12:46 | |
| *** leifmadsen has quit IRC | 12:48 | |
| *** leifmadsen has joined #openstack-infra | 12:48 | |
| openstackgerrit | Julien Danjou proposed a change to openstack-dev/hacking: Add a check for old style class declarations https://review.openstack.org/116846 | 12:49 |
| *** zz_dimtruck is now known as dimtruck | 12:51 | |
| *** pdmars has joined #openstack-infra | 12:53 | |
| *** jgallard has quit IRC | 12:54 | |
| *** pdmars has joined #openstack-infra | 12:54 | |
| *** sdake_ has quit IRC | 12:54 | |
| *** jgallard has joined #openstack-infra | 12:54 | |
| *** sdake_ has joined #openstack-infra | 12:55 | |
| *** marun has joined #openstack-infra | 12:55 | |
| openstackgerrit | Salvatore Orlando proposed a change to openstack-infra/elastic-recheck: iAdd footprint for bug 1362538 https://review.openstack.org/117468 | 12:55 |
| uvirtbot` | Launchpad bug 1362538 in bluez "bluetoothd crashed with SIGSEGV in server_disconnect()" [Medium,New] https://launchpad.net/bugs/1362538 | 12:55 |
| fungi | sdague: it's hand-managed due to checken&egg bootstrapping issues (it manages how you get access to update it) | 12:56 |
| fungi | sdague: though we have it basically documented at http://ci.openstack.org/gerrit.html#access-controls | 12:57 |
| fungi | s/checken/chicken/ | 12:57 |
| *** mfer has joined #openstack-infra | 12:57 | |
| *** ominakov has joined #openstack-infra | 12:57 | |
| *** joools has joined #openstack-infra | 12:58 | |
| *** dmakogon_ is now known as denis_makogon | 13:01 | |
| fungi | as for gate/check pipeline length, yeah it looks pretty bad but we've seen worse. hpcloud outage seems to have recovered many hours ago, though still could account for some of the backup there | 13:02 |
| fungi | mostly just lack of sufficient resources to run the pending jobs | 13:03 |
| fungi | but we are still pumping stuff through and running around 800 jobs an hour, which is about what we've been topping out at for weeks now | 13:03 |
| sdague | fungi: so could we enable Rebase for everyone? | 13:07 |
| fungi | and rax-dfw is pretty much a complete loss lately (seems like every time they unstick the error nodes we fill it right back up with them in a matter of a few hours) | 13:07 |
| fungi | sdague: enable the rebase button you mean? | 13:07 |
| fungi | (or api call, i suppose, same thing) | 13:07 |
| *** yfried_ has quit IRC | 13:07 | |
| fungi | we could probably switch that to registered users. might want to think about the potential downsides, but it seems doable anyway | 13:08 |
| sdague | fungi: yeh, enable the rebase button | 13:08 |
| fungi | sdague: or is that something we should just enable for project owner and core reviewer group on a project? | 13:09 |
| *** paulrad has joined #openstack-infra | 13:09 | |
| fungi | (like we do with wip, abandon/restore, et cetera) | 13:09 |
| *** jaypipes has joined #openstack-infra | 13:09 | |
| *** nkinder has quit IRC | 13:09 | |
| *** k4n0 has quit IRC | 13:10 | |
| *** sdake_ has quit IRC | 13:11 | |
| bookwar | hi all, i have "WARNING: The Gerrit-trigger send commands queue contains 44 items! Something might be stuck, or your system can't process the commands fast enough." in our third-third-party CI and I see review.openstack.org being lazy to respond today, are there any known issues or should I go debug more? | 13:12 |
| sdake | morning | 13:12 |
| fungi | bookwar: it probably has nothing to do with gerrit, but i'll check its inter-jvm process queue | 13:13 |
| fungi | bookwar: i don't see any stuck queues internally in gerrit, so it should be responding to your api commands in a timely manner | 13:14 |
| *** markmcclain has joined #openstack-infra | 13:14 | |
| sdague | fungi: so anyone can do it manually | 13:14 |
| fungi | sdague: that is very true | 13:15 |
| sdague | so it seems like making it a registered users permission is just giving people nicer ux | 13:15 |
| sdague | much like the commit message editor | 13:15 |
| fungi | though it takes some effort, whereas adding an easy button for everybody might become an attractive nuisance/free-for-all | 13:15 |
| sdague | maybe | 13:15 |
| *** sballe_ has quit IRC | 13:16 | |
| fungi | i can imagine some first-time reviewer deciding to go through and punch the rebase button on every change they see with a merge conflict message | 13:16 |
| *** gargola has joined #openstack-infra | 13:16 | |
| bookwar | fungi: thanks | 13:16 |
| fungi | without inspecting the results | 13:16 |
| *** sballe_ has joined #openstack-infra | 13:16 | |
| *** afazekas has joined #openstack-infra | 13:16 | |
| fungi | (keep in mind that rebases can go horribly wrong and act like they succeeded trivially, readding everyone's votes, so it's necessary to look at the end result carefully to make sure it did what you expect, which implies someone actually knowing what to expect/look for) | 13:17 |
| fungi | sdague: in fact, did you see https://review.openstack.org/117375 from last night? that one was downright scary (at merge time instead of a rebase, so nobody had an opportunity to notice it until after it went into production). thankfully the outcome was reasonably benign | 13:19 |
| *** emagana has joined #openstack-infra | 13:19 | |
| sdague | fungi: nope | 13:19 |
| *** dustins has joined #openstack-infra | 13:20 | |
| *** oomichi has quit IRC | 13:20 | |
| sdague | though most of the time we've got more unit tests on output on things, so that's a pretty far edge case | 13:21 |
| *** mfer has quit IRC | 13:23 | |
| Shrews | Any status update on https://bugs.launchpad.net/horizon/+bug/1362325? Safe to do rechecks yet? | 13:24 |
| uvirtbot` | Launchpad bug 1362325 in horizon "Dashboard with slug "sahara" is not registered" [Undecided,Confirmed] | 13:24 |
| *** emagana has quit IRC | 13:24 | |
| fungi | Shrews: yeah, i believe jeblair squashed SergeyLukjanov's two fixes into one last night abd crammed them up to the head of the gate | 13:24 |
| Shrews | fungi: cool. thx | 13:24 |
| *** radez_g0` is now known as radez | 13:24 | |
| SergeyLukjanov | fungi, Shrews, yup, it' fixed already, thx for jeblair | 13:25 |
| *** mriedem has joined #openstack-infra | 13:26 | |
| *** tomoe_ has joined #openstack-infra | 13:27 | |
| *** hdd has quit IRC | 13:28 | |
| *** dangers_away is now known as dangers | 13:30 | |
| *** dizquierdo_afk is now known as dizquierdo | 13:33 | |
| *** hashar_ has joined #openstack-infra | 13:34 | |
| *** hashar has quit IRC | 13:37 | |
| *** jgallard has quit IRC | 13:37 | |
| *** tomoe_ has quit IRC | 13:38 | |
| *** jgallard has joined #openstack-infra | 13:38 | |
| *** joools has quit IRC | 13:38 | |
| *** tomoe_ has joined #openstack-infra | 13:40 | |
| *** emagana has joined #openstack-infra | 13:40 | |
| *** mmaglana has joined #openstack-infra | 13:40 | |
| *** adalbas has quit IRC | 13:40 | |
| phschwartz | morning infra | 13:41 |
| phschwartz | fungi: we cleaned up dfw a second time yesterday so you should be good for now. | 13:41 |
| *** baohua has joined #openstack-infra | 13:42 | |
| fungi | phschwartz: you would think that, but i count 122 in error+deleting state in dfw right now | 13:42 |
| phschwartz | grrrr. stupid neutron. lol | 13:43 |
| fungi | seems we accumulate a quota full of those, roughly, within a few hours of instance boot/use/delete churn | 13:43 |
| *** salv-orlando has quit IRC | 13:43 | |
| phschwartz | very soon we will be able to turn the delete retry on for them and it will auto clean them up | 13:44 |
| fungi | keen | 13:44 |
| *** pblaho is now known as pblaho|afk | 13:45 | |
| *** dustins has quit IRC | 13:45 | |
| *** mmaglana has quit IRC | 13:45 | |
| *** afazekas has quit IRC | 13:46 | |
| *** doug-fish has joined #openstack-infra | 13:48 | |
| *** hashar_ is now known as hashar | 13:49 | |
| *** dkranz has joined #openstack-infra | 13:49 | |
| *** amotoki has quit IRC | 13:51 | |
| sdague | fungi: the f20 node mix seems to be problematic again as well | 13:51 |
| sdague | that's the reason some of the devstack changes have 20 hr delays on them | 13:51 |
| *** amotoki has joined #openstack-infra | 13:51 | |
| fungi | sdague: yeah, it becomes so as soon as we get beyond having enough capacity to service all job requests | 13:51 |
| *** Sincler has joined #openstack-infra | 13:52 | |
| sdague | so.... given that's not going to change for weeks, can we drop it again | 13:52 |
| sdague | I really don't like nv jobs blocking real fixes | 13:52 |
| *** adalbas has joined #openstack-infra | 13:52 | |
| fungi | sdague: well, the same effect is also happening to anyone running py33 jobs, pypy jobs, or anything still explicitly tied to ubuntu precise. basically any job that doesn't run on devstack-trusty or bare-trusty nodes (and to a slightly lesser extent, bare-centos6) is going to suffer this | 13:53 |
| sdague | yep | 13:53 |
| fungi | so, sure, we can make a judgement call that those aren't worth testing after all and are merely a nice-to-have when things are working smoothly. it's not really my call to make | 13:54 |
| sdague | well I let the f20 job back on devstack based on the fact that the allocator was fixed, but that seemed to have gotten reverted for bugs | 13:54 |
| *** homeless has joined #openstack-infra | 13:54 | |
| fungi | if that's the consensus among the devstack core reviewers, we can stop running it for now | 13:54 |
| fungi | yep, it turns out that it had bugs | 13:55 |
| fungi | ianw is still working to reproduce the condition under volume | 13:55 |
| *** stevemar has joined #openstack-infra | 13:56 | |
| fungi | speaking of bugs, i'm having some serious trouble tracking down the cause of http://logs.openstack.org/69/108269/17/check/check-tempest-dsvm-postgres-full/37ca571/console.html#_2014-08-28_09_45_28_460 | 13:57 |
| *** emagana has quit IRC | 13:57 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: Revert "Revert "demote f20 to experimental"" https://review.openstack.org/117518 | 13:57 |
| *** emagana has joined #openstack-infra | 13:58 | |
| fungi | i mean, http://logs.openstack.org/69/108269/17/check/check-tempest-dsvm-postgres-full/37ca571/logs/screen-c-api.txt.gz?level=TRACE is going on, which is supposed to have been fixed already four weeks ago | 13:58 |
| fungi | but i'm assuming that's just benign noise | 13:58 |
| *** koolhead17 has joined #openstack-infra | 13:58 | |
| *** pcrews has joined #openstack-infra | 13:59 | |
| *** pradk has joined #openstack-infra | 13:59 | |
| fungi | at least bug 1348786 claims https://review.openstack.org/109714 fixed that | 13:59 |
| *** pblaho|afk is now known as pblaho | 13:59 | |
| uvirtbot` | Launchpad bug 1348786 in devstack "VolumeTypeNotFoundByName spamming cinder-api logs" [Undecided,Fix released] https://launchpad.net/bugs/1348786 | 13:59 |
| *** gokrokve has joined #openstack-infra | 14:00 | |
| *** emagana has quit IRC | 14:02 | |
| tsufiev | jeblair, hello! could you please take a look at this commit https://review.openstack.org/#/c/110325/ for adding a new repo to stackforge? It already has one +2, also you had reviewed it a week ago - it seems that everything is in place now | 14:02 |
| sdague | fungi: well I don't know, though it's a terrible test | 14:02 |
| *** ominakov has quit IRC | 14:03 | |
| *** andreaf has joined #openstack-infra | 14:03 | |
| *** ociuhandu has joined #openstack-infra | 14:03 | |
| fungi | looks like all that traceback spam crept back in on the 23rd according to logstash, so probably a regression of 1348786 | 14:03 |
| *** eharney has joined #openstack-infra | 14:03 | |
| *** ominakov has joined #openstack-infra | 14:03 | |
| *** nkinder has joined #openstack-infra | 14:04 | |
| *** sandywalsh has joined #openstack-infra | 14:04 | |
| fungi | and yeah, mostly successful runs, so benign | 14:04 |
| *** tomoe_ has quit IRC | 14:04 | |
| *** r-daneel has joined #openstack-infra | 14:05 | |
| sdague | it's an error in keystone / cinder client / or the test | 14:05 |
| sdague | no other part is in play | 14:05 |
| fungi | sdague: thanks, i'll poke around in keystone some | 14:05 |
| sdague | actually, the reason why it failed is the next line | 14:06 |
| sdague | there is an update endpoint test happening at the same time | 14:06 |
| sdague | 2014-08-28 09:45:29.184 | {1} tempest.api.identity.admin.v3.test_endpoints.EndPointsTestXML.test_update_endpoint [0.771050s] ... ok | 14:06 |
| fungi | ahh, concurrency-related? | 14:06 |
| sdague | yep | 14:06 |
| sdague | my guess at least | 14:06 |
| fungi | i would never have spotted that | 14:06 |
| fungi | it wasn't clicking for me that this was funneled through keystone as well | 14:07 |
| *** mmedvede has quit IRC | 14:08 | |
| *** annegent_ has joined #openstack-infra | 14:08 | |
| *** dustins has joined #openstack-infra | 14:08 | |
| sdague | yeh, the path for endpoints only goes to keystone as far as I know | 14:08 |
| *** chuckC has quit IRC | 14:08 | |
| *** emagana has joined #openstack-infra | 14:09 | |
| sdague | unfortunately the cli tests aren't written in a way that lets us see the entire catalog, they seem prematurely optimized for code reuse :{ | 14:10 |
| *** dimtruck is now known as zz_dimtruck | 14:10 | |
| sdague | mriedem: you played with the cli tests some right? | 14:11 |
| *** zz_dimtruck is now known as dimtruck | 14:11 | |
| *** mmedvede has joined #openstack-infra | 14:11 | |
| sdague | any idea how we'd change those to expose the whole catalog on those assert fails? | 14:11 |
| *** sdake_ has joined #openstack-infra | 14:14 | |
| *** andreaf has quit IRC | 14:14 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: Revert "Revert "demote f20 to experimental"" https://review.openstack.org/117518 | 14:14 |
| *** hdd has joined #openstack-infra | 14:15 | |
| mriedem | sdague: catching up, you mean dump out the keystone service catalog on any cli failure? | 14:15 |
| sdague | well all the comparisons are so hyper normalized I don't know what it's doing | 14:15 |
| *** dmsimard_away is now known as dmsimard | 14:15 | |
| mriedem | which comparisons? | 14:15 |
| sdague | the error seems non sensible | 14:15 |
| sdague | http://logs.openstack.org/69/108269/17/check/check-tempest-dsvm-postgres-full/37ca571/console.html#_2014-08-28_10_04_15_757 | 14:16 |
| sdague | I would expect to see more than 1 catalog line | 14:16 |
| sdague | MismatchError: 'nova' not in {'type---446510413': 'id', 'Value': '0900b1eaee174fdd9e146438a25b845e'} | 14:16 |
| sdague | it seems like endpoints is actually only the 1st endpoint | 14:17 |
| *** mmedvede has quit IRC | 14:17 | |
| mriedem | checking the tempest code | 14:17 |
| jeblair | sdague: https://review.openstack.org/#/c/113812/ may have been caught by the freshness check | 14:17 |
| *** armax has joined #openstack-infra | 14:17 | |
| jeblair | https://jenkins07.openstack.org/job/gate-nova-python27/5840/console https://jenkins03.openstack.org/job/gate-nova-python26/4710/console | 14:18 |
| *** mmedvede has joined #openstack-infra | 14:18 | |
| sdague | jeblair: yeh | 14:18 |
| jeblair | both unit tests jobs failed; last check was 3 days ago | 14:18 |
| *** david-lyle has joined #openstack-infra | 14:19 | |
| mriedem | sdague: so you expected to see the cinder v1 and v2 endpoints? | 14:19 |
| jeblair | i don't suppse anyone got around to writing a dequeue command :/ | 14:19 |
| sdague | mriedem: that's not actually what it is | 14:19 |
| sdague | jeblair: edit commit message in gerrit? | 14:19 |
| sdague | that's what I do | 14:19 |
| sdague | much easier now that you can do that via gerrit web | 14:20 |
| sdague | mriedem: so cinder endpoints actually returns the whole catalog | 14:21 |
| sdague | and the test seems to just care that nova is listed somewhere | 14:21 |
| sdague | except, what it actually seems to care about is that nova is listed first | 14:21 |
| sdague | so if something else ends up first | 14:21 |
| *** otherwiseguy has joined #openstack-infra | 14:21 | |
| fungi | that's an amusing test... for some values of amusing anyway | 14:21 |
| sdague | because endpoints are being added at the same time (which is happening) | 14:22 |
| sdague | then... fail | 14:22 |
| mriedem | sdague: oh, huh - why does a cinder endpoint list test care that nova is in there? | 14:23 |
| *** leifmadsen is now known as SgtPants | 14:23 | |
| sdague | because someone wrote it to just find something | 14:23 |
| sdague | I expect they copy/pasted the nova test | 14:23 |
| sdague | and it shows all endpoints | 14:23 |
| *** annegent_ has quit IRC | 14:23 | |
| sdague | so it should be fine ... except it's not actually looking at all the endpoints | 14:23 |
| *** annegent_ has joined #openstack-infra | 14:24 | |
| mriedem | sdague: is this a relatively new failure? | 14:24 |
| * mriedem is putting out an internal fire this morning so is behind | 14:24 | |
| *** Longgeek_ has quit IRC | 14:24 | |
| *** SgtPants is now known as leifmadsen | 14:24 | |
| *** jgrimm_zz is now known as jgrimm | 14:24 | |
| sdague | it was something fungi was looking at | 14:25 |
| sdague | it was in the last 24hrs | 14:25 |
| *** _nadya_ has joined #openstack-infra | 14:25 | |
| fungi | yeah, random test failure i couldn't find any obvious fingerprint for | 14:25 |
| sdague | mriedem: no worries if there are other fires | 14:25 |
| fungi | and for which i couldn't find a bug either | 14:25 |
| fungi | presumably low-incidence so far | 14:26 |
| sdague | I think it's just indicative of the fact that the cli tests really need to come out of tempest, they make some terrible assumptions :) | 14:26 |
| mriedem | sdague: fungi: according to logstash it's only failed on that one pbr change | 14:28 |
| fungi | jeblair: oh, on the topic of the nodepool 'splosions during oscon which prompted us to revert the demand fulfillment algorithm change, did you happen to set aside a copy of the nodepool logs from that time? we're just a few days past log rotation having expired them out of /var | 14:28 |
| fungi | mriedem: yeah, in that case *very* low incidence | 14:28 |
| jeblair | fungi: yeah, i'm planning on working on that today | 14:28 |
| *** andreaf has joined #openstack-infra | 14:29 | |
| jeblair | ianw: i mentioned i'd be able to pitch in later this week; i'm all yours :) | 14:29 |
| *** yamahata has quit IRC | 14:29 | |
| *** jistr has quit IRC | 14:29 | |
| fungi | jeblair: okay, awesome. i went looking for them earlier this week so i could sanitize a copy for ianw but discovered they were already gone | 14:29 |
| jeblair | assuming that the blowup happened on 7/23, i have copies of logs in my homedir | 14:29 |
| fungi | jeblair: yep, i hadn't gone poking anywhere else obvious yet besides /var, but that was the timeframe i was looking for (it coincides with the revert commit) | 14:30 |
| *** david-lyle has quit IRC | 14:30 | |
| *** salv-orlando has joined #openstack-infra | 14:30 | |
| *** david-lyle has joined #openstack-infra | 14:30 | |
| sdague | mriedem: yeh, it's probably infrequent because the keystone tests are fast | 14:31 |
| sdague | so crosssection for overlap is small | 14:31 |
| *** markmcclain has quit IRC | 14:32 | |
| jeblair | fungi, ianw: yep, 3 restarts in the log on 7/23 | 14:33 |
| jeblair | i'll concentrate on the middle restart | 14:33 |
| *** sdake_ has quit IRC | 14:33 | |
| mriedem | sdague: fungi: so do we want to open a tempest bug for this? assuming 'yes' even though it has only shown up in one change in 7 days | 14:33 |
| *** juzuluag has joined #openstack-infra | 14:34 | |
| dtroyer | fungi: https://review.openstack.org/117522 should fix the Cinder volume type regression | 14:34 |
| *** annegent_ has quit IRC | 14:34 | |
| *** masayukig has quit IRC | 14:34 | |
| *** mmaglana has joined #openstack-infra | 14:37 | |
| fungi | mriedem: yup, will do | 14:37 |
| fungi | dtroyer: oh, cool | 14:37 |
| *** masayukig has joined #openstack-infra | 14:40 | |
| *** dragondm has joined #openstack-infra | 14:40 | |
| *** rainya has joined #openstack-infra | 14:40 | |
| *** _nadya_ has quit IRC | 14:40 | |
| yolanda_ | jeblair, wrote a comment for https://review.openstack.org/#/c/105931/. The patch isn't needed now | 14:40 |
| HeOS | fungi, hello! | 14:41 |
| *** hashar has quit IRC | 14:41 | |
| HeOS | fungi, do you known directive inheritFrom = <parent_project_name> in project.config is worked in jeepyb? | 14:42 |
| *** tnurlygayanov has quit IRC | 14:42 | |
| *** andreaf has quit IRC | 14:43 | |
| *** dkehn__ has joined #openstack-infra | 14:43 | |
| *** andreaf has joined #openstack-infra | 14:43 | |
| HeOS | I received an error ! [remote rejected] HEAD -> refs/meta/config (invalid project configuration: parent does not exist), but <parent_project_name> is exist in gerrit. | 14:43 |
| jeblair | fungi, ianw: looking at the logs more closely, it seems to be pass #3 and greater that end up with the radical over allocations | 14:43 |
| jeblair | (not so much #2 as we previously surmised) | 14:44 |
| openstackgerrit | Ken Giusti proposed a change to openstack-infra/config: Add a test to verify oslo.messaging's AMQP 1.0 messaging protocol support https://review.openstack.org/115752 | 14:45 |
| *** trinaths has joined #openstack-infra | 14:45 | |
| *** wenlock has joined #openstack-infra | 14:45 | |
| *** dkehn_ has quit IRC | 14:45 | |
| *** atiwari has joined #openstack-infra | 14:45 | |
| *** dkehn has quit IRC | 14:46 | |
| anteaya | jesusaurus: thanks, I'm also looking for open formats, is mp4 and ac3 or acc open? I don't know, I haven't spent time in this space lately as a creator | 14:46 |
| anteaya | nibalizer: yes, I find ffmpeg overwhelming but would like to spend some time with it | 14:46 |
| HeOS | In project.config file only next two lines: 1: [access] 2: inheritFrom = <parent_project_name> | 14:46 |
| *** hashar has joined #openstack-infra | 14:46 | |
| *** bradjones has quit IRC | 14:46 | |
| jroll | anteaya: flac and ogg are really the only open formats afaik | 14:47 |
| jroll | or is this video? | 14:47 |
| *** dkehn has joined #openstack-infra | 14:47 | |
| *** yolanda_ has quit IRC | 14:47 | |
| jroll | yeah, misread that, ignore me :) | 14:47 |
| *** yolanda has joined #openstack-infra | 14:49 | |
| jeblair | fungi: nodepool.o.o:~corvus/log | 14:49 |
| *** annegent_ has joined #openstack-infra | 14:49 | |
| jeblair | fungi: i've extracted the section that deals with startup #2, and checked that none of our defined passwords show up in there | 14:49 |
| jeblair | fungi: anything else you want to check? | 14:49 |
| anteaya | jroll: video and audio | 14:50 |
| anteaya | lyxus trinaths this is normal load for where we are in the release cycle: https://wiki.openstack.org/wiki/Juno_Release_Schedule | 14:50 |
| *** zehicle_at_dell has quit IRC | 14:51 | |
| anteaya | lyxus trinaths as you see next week is feature freeze and so we are under normal load (meaning extreme) since everyone wants their patches merged before Sept 4 | 14:51 |
| *** ildikov has joined #openstack-infra | 14:51 | |
| jroll | anteaya: http://en.wikipedia.org/wiki/List_of_open-source_codecs#Video_codecs :) | 14:51 |
| *** tonytan4ever has joined #openstack-infra | 14:52 | |
| anteaya | lyxus trinaths this is also the time of cycle we discover new and exciting things about our automating testing struture since we are under heavy load for a long time | 14:52 |
| *** bradjones has joined #openstack-infra | 14:52 | |
| anteaya | jroll: thanks | 14:52 |
| *** bradjones has quit IRC | 14:52 | |
| *** bradjones has joined #openstack-infra | 14:52 | |
| *** joools has joined #openstack-infra | 14:53 | |
| *** emagana has quit IRC | 14:53 | |
| *** jcoufal has quit IRC | 14:53 | |
| phschwartz | fungi: we are going to try and get them cleaned up again durring the day today, and at that time have a member of our internal neutron team watch as they build up again to see if it can help them find the actual cause. | 14:54 |
| *** andreaf has quit IRC | 14:54 | |
| *** nelsnelson has quit IRC | 14:54 | |
| *** andreaf has joined #openstack-infra | 14:54 | |
| jeblair | phschwartz: we'll, uh, make sure you have enough load to detect the problem. ;) | 14:55 |
| *** chuckC has joined #openstack-infra | 14:55 | |
| fungi | phschwartz: yeah, i think there'll be no shortage of incidents for them to analyze there | 14:57 |
| fungi | jeblair: looking | 14:57 |
| *** mmaglana has quit IRC | 14:57 | |
| *** _nadya_ has joined #openstack-infra | 14:58 | |
| sdague | jeblair: so I've been going through my review list, man would auto abandon help. | 14:58 |
| *** skolekonov has quit IRC | 14:58 | |
| jeblair | sdague: it would really hurt mine. we have reviews for big efforts that sit for weeks. they should not be abandoned. | 14:59 |
| jeblair | sdague: for instance, there's no way we can make significant progress on nodepool dib with the queue in its current state. do we abandon all the reviews? no, we just let them sit until things cool down and pick it up again later | 14:59 |
| *** kmartin has joined #openstack-infra | 15:00 | |
| jeblair | sdague: we just merged a review number that was _half_ of the current review number | 15:00 |
| sdague | so the pattern I'm seeing, fwiw, as I've been through about 100 old reviews in the last 24 hrs | 15:00 |
| sdague | is that developers that are highly engaged in the project seem to have a good sense of self abandoning code | 15:01 |
| *** jistr has joined #openstack-infra | 15:01 | |
| jeblair | sdague: why? had you not previously reviewed them? | 15:01 |
| sdague | and when it's appropriate | 15:01 |
| jeblair | oh i misunderstood that sentence | 15:01 |
| *** balles has joined #openstack-infra | 15:01 | |
| *** balles has quit IRC | 15:01 | |
| sdague | jeblair: yeh, sorry, these were reviews that I was already listed as a reviewer on | 15:01 |
| sdague | some with last activity going to back to May | 15:01 |
| openstackgerrit | Mike Heald proposed a change to openstack-infra/jenkins-job-builder: Add support for scripts in the inject builder https://review.openstack.org/117528 | 15:02 |
| sdague | however people less connected to the project tend to leave -2ed or merge conflicted code out there | 15:02 |
| jeblair | sdague: my question is, why are you seeing reviews you don't want to? why don't you add "age:2w" to your queries? you'd get the same behavior as before | 15:02 |
| sdague | I'm going through the My Changes list just to clean it up a bit | 15:03 |
| sdague | also, it was mostly an exercise to understand what's up with these really old reviews | 15:04 |
| openstackgerrit | Romain Chantereau proposed a change to openstack-infra/config: Be able to generate math formulae with Sphinx mathpng base extension. https://review.openstack.org/117263 | 15:04 |
| sdague | so both cleanup, and exploration | 15:04 |
| jeblair | i'm guessing that we can't add "age:2w" to my changes.. but to be honest, i thought we had all written 'my changes' off as a useful part of the ui | 15:05 |
| jeblair | i mean, we did that 3 years ago when we wrote 'important changes' | 15:05 |
| sdague | important changes doesn't work for me | 15:05 |
| sdague | for volume reasons | 15:05 |
| jeblair | sdague: it doesn't work for anyone, it's gone | 15:05 |
| sdague | ok, fair | 15:06 |
| *** jcoufal-mobile has joined #openstack-infra | 15:06 | |
| *** jcoufal-mobile has quit IRC | 15:06 | |
| jeblair | sdague: but my changes had all of those problems and more | 15:06 |
| jeblair | so i'm surprised to hear that you use it | 15:06 |
| sdague | yeh, I don't use it all the time, I do come through it some times | 15:07 |
| *** coolsvap has quit IRC | 15:07 | |
| *** sdake_ has joined #openstack-infra | 15:07 | |
| sdague | I definitely use it for outgoing | 15:07 |
| jeblair | oh yeah | 15:07 |
| jeblair | right now that's long enough for me i can't actually see the incoming ones :) | 15:07 |
| sdague | yeh, so when that happens I typically stop writing code | 15:07 |
| sdague | because I can't keep that many contexts running in parallel | 15:08 |
| sdague | then I go and bug people to figure out why some things aren't landing | 15:08 |
| *** zz_gondoi is now known as gondoi | 15:09 | |
| sdague | anteaya / krtaylor: does someone want to take over this recheck gramar proposal - https://review.openstack.org/#/c/109565/ - because I'm not really going to be the right person to drive that to some concensus | 15:12 |
| anteaya | sdague: yeah dougwig is trying to do something with it | 15:13 |
| krtaylor | sdague, sure | 15:13 |
| dhellmann | hi, folks. I'd like to get oslo.log imported so we can cut stable/juno in oslo-incubator and start dropping graduated modules. It only needs one more +2: https://review.openstack.org/#/c/116934/ | 15:13 |
| anteaya | sdague: if you want you can abandon it and dougwig can propose again | 15:13 |
| anteaya | sdague: can you leave a comment that you are abandoning it due to time issues and someone else is welcome to use your code? | 15:13 |
| *** salv-orlando has quit IRC | 15:14 | |
| *** datsun180b has joined #openstack-infra | 15:15 | |
| sdague | anteaya: sure | 15:16 |
| *** coolsvap has joined #openstack-infra | 15:16 | |
| anteaya | sdague: since basically what I am hearing you say is you want this patch off your stack | 15:17 |
| anteaya | which I understand | 15:18 |
| sdague | anteaya: yes. Well mostly I'm afraid people are expecting me to drive it forward | 15:18 |
| anteaya | right | 15:18 |
| sdague | which it's not going to really be a priority for me | 15:18 |
| * anteaya nods | 15:18 | |
| anteaya | yup | 15:18 |
| sdague | so I wanted the right signaling to let someone else run with it | 15:18 |
| sdague | I hit abandone | 15:19 |
| anteaya | I think that will do it | 15:19 |
| anteaya | dougwig is the only person so far willing to make any noise about it, hopefully he will champion it for a while | 15:19 |
| sdague | yeh, every group is different | 15:19 |
| anteaya | it is | 15:19 |
| sdague | so I wasn't sure what the right way to signal that was here | 15:19 |
| *** mwagner_lap has joined #openstack-infra | 15:20 | |
| anteaya | yes, let's find out | 15:20 |
| anteaya | I'm trying to find ways to not be so tired by everything | 15:20 |
| anteaya | the non-stop construction 2 doors down isn't helping | 15:20 |
| krtaylor | anteaya, sdague I can help dougwig with moving that along if needed | 15:20 |
| krtaylor | anteaya, I feel your pain, next-door is getting new roof | 15:21 |
| anteaya | krtaylor: great it is all yours | 15:22 |
| anteaya | krtaylor: propose a new patch with yourself as owner and reference sdague's abandoned patch in the commit message, and list sdague as co-author in the commit message too | 15:23 |
| anteaya | Co-Authored By: Sean Dague <sean's email goes here> | 15:24 |
| anteaya | sdague: what email? | 15:24 |
| krtaylor | anteaya, I should check with dougwig first? | 15:24 |
| sdague | sean@dague.net is what I use | 15:24 |
| openstackgerrit | Steve Martinelli proposed a change to openstack-infra/config: Add readthedocs for heat-translator https://review.openstack.org/117533 | 15:24 |
| anteaya | sdague: great thanks | 15:24 |
| anteaya | krtaylor: up to you | 15:25 |
| dougwig | krtaylor: go for it. | 15:25 |
| sdague | thanks anteaya / krtaylor | 15:25 |
| *** mmaglana has joined #openstack-infra | 15:25 | |
| anteaya | thanks dougwig | 15:25 |
| dougwig | on phone, hard to type , let me know if I can help. | 15:25 |
| anteaya | then dougwig gets to be your first reviewer | 15:25 |
| jeblair | fungi: how's that log looking? | 15:25 |
| *** emagana has joined #openstack-infra | 15:26 | |
| *** signed8bit has joined #openstack-infra | 15:26 | |
| fungi | jeblair: i'm continuing to whittle and summarize | 15:26 |
| jeblair | fungi: oh i thought you just wanted to sanitize; i don't see why we can't serve up the whole thing | 15:27 |
| krtaylor | anteaya, dougwig - I'll get that up again this afternoon | 15:27 |
| fungi | jeblair: oh, we probably can... i was actually trying to analyze the content! sorry | 15:27 |
| jeblair | fungi: yeah, i'm all for that, but i figured first we can let ianw see it :) | 15:27 |
| *** ominakov has quit IRC | 15:28 | |
| fungi | absolutely. lemme make sure there's nothing leaked in there | 15:28 |
| *** HenryG has quit IRC | 15:28 | |
| *** ominakov has joined #openstack-infra | 15:29 | |
| *** _buzz_ has joined #openstack-infra | 15:32 | |
| *** annegent_ has quit IRC | 15:32 | |
| jeblair | fungi: 1600 utc saturday good for the glance.store rename? | 15:32 |
| fungi | jeblair: i think it should be safe. i scoured it for any credentials nodepool could possibly have access (ssh private key, jenkins credentials id, jenkins api password, database password) | 15:33 |
| jeblair | if so, i'll go ahead and send an announcement | 15:33 |
| fungi | jeblair: sure, 1600 utc works fine for me | 15:33 |
| *** IvanBerezovskiy has left #openstack-infra | 15:33 | |
| jeblair | considering the backlog, it might not be as quiet as i'd hoped, but we can re-enqueue | 15:33 |
| fungi | oh, also checked the log for the cloud provider api credentials as well | 15:33 |
| fungi | so, yeah, nothing i can find which looks sensitive in there | 15:34 |
| *** _buzz has quit IRC | 15:35 | |
| *** sdake_ has quit IRC | 15:35 | |
| *** habib has joined #openstack-infra | 15:35 | |
| jeblair | gerrit downtime announcement sent | 15:36 |
| *** markmcclain has joined #openstack-infra | 15:37 | |
| *** gokrokve_ has joined #openstack-infra | 15:38 | |
| sdague | jeblair: I guess you didn't kick out https://review.openstack.org/#/c/113812/ ? | 15:40 |
| sdague | I just did with a commit message change as I found it was back into a fail state | 15:40 |
| *** baohua has quit IRC | 15:41 | |
| *** gokrokve has quit IRC | 15:41 | |
| *** annegent_ has joined #openstack-infra | 15:42 | |
| jeblair | sdague: ok thanks. at the time, it had only failed once in gate, even though both jobs failed. i was on the fence about it. | 15:43 |
| jeblair | sdague: but yes, it seems pretty reliable | 15:44 |
| *** reed has joined #openstack-infra | 15:45 | |
| *** gyee has joined #openstack-infra | 15:47 | |
| *** annegent_ has quit IRC | 15:47 | |
| stevemar | so if I add docs-on-rtfd to a project's template, the readthedocs does some magic behind the scenes to try and make the theme rtfd-ish | 15:48 |
| *** dkehn__ is now known as dkehnx | 15:48 | |
| *** david-ly_ has joined #openstack-infra | 15:48 | |
| *** trinaths has quit IRC | 15:50 | |
| *** mfer has joined #openstack-infra | 15:50 | |
| *** koolhead17 has quit IRC | 15:51 | |
| *** david-lyle has quit IRC | 15:51 | |
| *** hashar has quit IRC | 15:51 | |
| fungi | jeblair: ianw: after some machete work, these are the log line patterns and their relative counts i find in that log sample related to allocation request fulfillment calculations http://paste.openstack.org/show/101805/ | 15:55 |
| *** todd_dsm has joined #openstack-infra | 15:55 | |
| fungi | opening some of the N values there and summing them will probably yield the trend we witnessed | 15:55 |
| *** pblaho is now known as pblaho|afk | 15:56 | |
| jeblair | ianw: http://nodepool.openstack.org/allocator-failure.log | 15:57 |
| openstackgerrit | Steve Martinelli proposed a change to openstack-infra/config: Add readthedocs for heat-translator https://review.openstack.org/117533 | 16:01 |
| sdague | jeblair: yeh, honestly, I tend to err on the kick it out side once there is reasonable indicators, because otherwise these add 30 - 60 minutes delay to every patch behind them. | 16:01 |
| *** annegent_ has joined #openstack-infra | 16:01 | |
| *** amuller_ has joined #openstack-infra | 16:02 | |
| sdague | jeblair: https://review.openstack.org/#/c/114991/ is another one that the freshness check would have caught - ceilometer patch | 16:02 |
| openstackgerrit | Joel Coffman proposed a change to openstack/requirements: Add pykmip https://review.openstack.org/114037 | 16:03 |
| *** cody-somerville has joined #openstack-infra | 16:03 | |
| *** cody-somerville has quit IRC | 16:03 | |
| *** cody-somerville has joined #openstack-infra | 16:03 | |
| *** jgallard has quit IRC | 16:03 | |
| sdague | double unit test failures is a pretty good initial indicator, that case is importing some olso code that doesn't currently exist | 16:03 |
| *** amuller has quit IRC | 16:04 | |
| *** reed has quit IRC | 16:04 | |
| jeblair | sdague: yeah, and that sat for a long time, 9 days | 16:05 |
| *** markmc has quit IRC | 16:05 | |
| sdague | yep | 16:05 |
| sdague | which is about as long as it could, right? | 16:05 |
| sdague | because that's around when the freshness was turned off | 16:05 |
| *** melwitt has joined #openstack-infra | 16:06 | |
| jeblair | sdague: that very day in fact | 16:06 |
| openstackgerrit | Joel Coffman proposed a change to openstack/requirements: Add pykmip https://review.openstack.org/114037 | 16:06 |
| sdague | right | 16:06 |
| *** jraim__ has quit IRC | 16:06 | |
| sdague | so we are definitely getting changes that are as old on results as they could possibly by being approved in | 16:06 |
| *** linkid has left #openstack-infra | 16:06 | |
| *** linkid has joined #openstack-infra | 16:06 | |
| *** jraim_ has joined #openstack-infra | 16:06 | |
| jeblair | sdague: i'm wondering - if we decide the arguments favor gate freshness being worth the cost, if a different time cutoff would work; perhaps 48 or 72 hours would get us the bulk of the value. | 16:08 |
| sdague | jeblair: yeh, I don't know | 16:08 |
| *** dustins has quit IRC | 16:08 | |
| anteaya | what was the argument for turning it off? | 16:08 |
| sdague | if we are only requiring the final threshold for tests moving to gate, it seems like it's O(1) cost, so making it 24 hrs doesn't seem more expensive than 72 | 16:09 |
| sdague | I think leaving the always fresh test results off is fine | 16:09 |
| *** habib has quit IRC | 16:10 | |
| sdague | anyway, just a thought. I'm going to be in and out for the rest of the afternoon. | 16:11 |
| *** paulrad has quit IRC | 16:11 | |
| *** annegent_ has quit IRC | 16:12 | |
| *** trinaths has joined #openstack-infra | 16:12 | |
| *** melwitt has quit IRC | 16:12 | |
| *** melwitt has joined #openstack-infra | 16:12 | |
| *** e0ne has quit IRC | 16:12 | |
| openstackgerrit | yolanda.robla proposed a change to openstack-infra/publications: New CI Automation doc https://review.openstack.org/109570 | 16:15 |
| *** zehicle_at_dell has joined #openstack-infra | 16:15 | |
| morganfainberg | jeblair, [if i'm reading this right] i agree with sdague, if it's only on moving to gate we get the freshness requirement it would be (relatively) low cost compared to gate resets. The added cost of forcing a freshness check based on a comment but not a move to gate seems like extra overhead. | 16:17 |
| *** lukego has quit IRC | 16:17 | |
| morganfainberg | jeblair, at least around milestone timeframes. | 16:17 |
| *** nelsnelson has joined #openstack-infra | 16:17 | |
| *** isviridov is now known as isviridov_away | 16:18 | |
| *** nelsnelson has quit IRC | 16:18 | |
| jeblair | morganfainberg: yeah, the freshness check in check on comment wasn't really about comments, it was just an implementation mechanism because we hadn't written anything else. it was really about collecting data and having up-to-date results | 16:18 |
| jeblair | iow, it was more about the 72h than the comment. but your point still stands; just clarifying. | 16:18 |
| morganfainberg | jeblair, ++ figured as much | 16:18 |
| *** nelsnelson has joined #openstack-infra | 16:18 | |
| *** Ryan_Lane has quit IRC | 16:19 | |
| jeblair | i think we were using up to 3000 nodes/day on that; i forget the number used for gate. it was far smaller (tens or hundreds), but that would probably be increased if we put the gate freshness check back without the check one | 16:20 |
| *** pballand has joined #openstack-infra | 16:20 | |
| *** dguerri_ is now known as dguerri | 16:20 | |
| morganfainberg | jeblair, yeah 3000 is a lot of extra nodes for that. | 16:21 |
| *** annegent_ has joined #openstack-infra | 16:21 | |
| morganfainberg | jeblair, 72h is probably fine in most cases, not sure if it would be a bigger benefit closer to 24h for milestone timeframes. i saw a marked increase errors that cropped up in keystone on checks when we started with large numbers of things going through the gate | 16:22 |
| morganfainberg | jeblair, at least the merge conflicts seemed to be caught earlier | 16:23 |
| *** nelsnelson has quit IRC | 16:23 | |
| *** todd_dsm has quit IRC | 16:23 | |
| *** nelsnelson has joined #openstack-infra | 16:23 | |
| jeblair | morganfainberg: yeah, that was probably the biggest benefit of the check-pipeline freshness checks, and it's replaced by the merge-check pipeline now... | 16:24 |
| jeblair | morganfainberg: merge conflicts have been low-cost in the gate for a while | 16:24 |
| fungi | which is a much faster response time on reporting merge conflicts than waiting for activity on the change before checking | 16:24 |
| morganfainberg | jeblair, ++ and i totally approve of that change :) | 16:24 |
| *** joools has quit IRC | 16:25 | |
| jeblair | our informal spot check has caught 2 changes in the gate that would have been caught by freshness | 16:25 |
| jeblair | they sat for 3 and 10 days without tests respectively | 16:25 |
| jeblair | of course, i don't know when the bugs that caused them to fail were introduced | 16:26 |
| jeblair | so it's kind of a statistics game, which is hard to play with 2 data points :) | 16:26 |
| jeblair | but my gut feeling is that 24<x<72 would give us much of the benefit | 16:27 |
| *** todd_dsm has joined #openstack-infra | 16:28 | |
| *** esker has joined #openstack-infra | 16:28 | |
| *** paulrad has joined #openstack-infra | 16:28 | |
| jeblair | i kicked out that ceilometer change (https://review.openstack.org/#/c/114991/) | 16:29 |
| *** david-ly_ is now known as david-lyle | 16:30 | |
| *** trinaths has quit IRC | 16:30 | |
| *** annegent_ has quit IRC | 16:30 | |
| *** jaypipes has quit IRC | 16:30 | |
| *** trinaths has joined #openstack-infra | 16:32 | |
| *** todd_dsm has quit IRC | 16:33 | |
| *** gokrokve has joined #openstack-infra | 16:33 | |
| notmyname | sdague: I have some questions about https://review.openstack.org/#/c/117216/. I don't understand why the version was bumped. it looks more like futures just wasn't installed on the test box (maybe because it's a new requirement?) but it works for me with the older version | 16:34 |
| *** _nadya_ has quit IRC | 16:34 | |
| notmyname | sdague: (if anything, I'd love to drop to 2.1.2 since that's what's available in precise, but that's a different topic) | 16:34 |
| *** pblaho|afk is now known as pblaho | 16:34 | |
| *** marcoemorais has joined #openstack-infra | 16:35 | |
| sdague | notmyname: the swiftclient add required concurrent.futures | 16:35 |
| *** derekh has quit IRC | 16:35 | |
| notmyname | sdague: which is a new module, provided by the "futures" package | 16:35 |
| sdague | notmyname: ok, so perhaps we misinterpreted the error message | 16:36 |
| *** gokrokve_ has quit IRC | 16:36 | |
| sdague | sorry, I guess the googling I popped up made me think it was a 2.1.6 feature | 16:36 |
| morganfainberg | jeblair, cannot complain about the logic, lgtm :) <72 sounds about right. | 16:36 |
| notmyname | sdague: https://gist.github.com/notmyname/5d20e347d208de2fcff0 | 16:36 |
| sdague | notmyname: ok, well previously it was set at 2.1.3 | 16:37 |
| sdague | notmyname: feel free to propose revert back to 2.1.3 | 16:37 |
| *** jpich has quit IRC | 16:37 | |
| notmyname | sdague: so consider this my vote to go down to 2.1.2 (it was set at 2.1.3 because that's what was already in global reqs) | 16:37 |
| fungi | mriedem: the bug we were talking about earlier is opened now as 1362703 | 16:37 |
| sdague | I guess I misinterpretted the failures | 16:37 |
| *** gokrokve has quit IRC | 16:37 | |
| sdague | notmyname: yeh, I don't know the reason why 2.1.3 was set as the floor | 16:38 |
| notmyname | sdague: does this mean there is another issue that needs to be set | 16:38 |
| notmyname | err...s/set/resolved | 16:38 |
| mriedem | fungi: thanks | 16:38 |
| notmyname | sdague: eg not installing the right dependencies? or that the new dependencies have to be added to the old stable branch | 16:38 |
| *** gokrokve has joined #openstack-infra | 16:38 | |
| sdague | well, we needed to add it to havana requirements because swiftclient could no long work on a havana install | 16:38 |
| *** ominakov has quit IRC | 16:39 | |
| sdague | anyway, I need to run, but the scrollback yesterday should have all the details in it. If someone wants to try to make this all work at 2.1.2 across projects, so be it. | 16:40 |
| *** ominakov has joined #openstack-infra | 16:40 | |
| *** _nadya_ has joined #openstack-infra | 16:40 | |
| notmyname | sdague: if I proposed a change to global requirements to use 2.1.2, woudln't that check it sufficiently? | 16:40 |
| notmyname | I suspect 2.1.3 was used because the first time is was added that was the current version | 16:40 |
| *** mwagner_lap has quit IRC | 16:41 | |
| *** clayg has joined #openstack-infra | 16:43 | |
| *** gokrokve has quit IRC | 16:43 | |
| *** andreaf_ has joined #openstack-infra | 16:43 | |
| *** eharney has quit IRC | 16:44 | |
| *** andreaf has quit IRC | 16:44 | |
| clarkb | notmyname: it would not check it sufficiently | 16:44 |
| notmyname | clarkb: what would? | 16:44 |
| clarkb | notmyname: apply that requireemtns line to all projects using the dependency and run their docs, pep8, unittest tests would help a lot more | 16:45 |
| clarkb | notmyname: since requirements changes don't do that and instead run integration tests which are good but may have gaps | 16:45 |
| notmyname | clarkb: but it has to land in global reqs first, riht? | 16:45 |
| *** jaypipes has joined #openstack-infra | 16:46 | |
| notmyname | otherwise tests fail | 16:46 |
| clarkb | notmyname: no | 16:46 |
| *** mpaolino has joined #openstack-infra | 16:46 | |
| * notmyname clearly doesn't understand what's going on | 16:46 | |
| clarkb | I mean yes tests fail but not because it can't import the dependency | 16:46 |
| *** jistr has quit IRC | 16:46 | |
| clarkb | there is a special test that checks, that one will fail but won't affect the unittests and other builds | 16:46 |
| *** Hal_ has quit IRC | 16:47 | |
| fungi | right, you can propose a throwaway change to a project to see how it would deal with setting the requirement to that version explicitly, and just ignore the requirements comparison job failure it will cause | 16:47 |
| *** esker has quit IRC | 16:48 | |
| *** esker has joined #openstack-infra | 16:48 | |
| notmyname | ok | 16:48 |
| notmyname | so why didn't grenade fail on the patch with the new dependency? https://review.openstack.org/#/c/85453/27 | 16:48 |
| fungi | that would actually try to run unit tests with the proposed version | 16:48 |
| *** _nadya_ has quit IRC | 16:49 | |
| fungi | grenade and other things using devstack will force-sync your list of requirements to the versions specified in the global requirements list, so the integration tests on individual project changes are not a good indicator | 16:49 |
| jeblair | perhaps grenade force-synced the master branch when it ran the old side of the job? | 16:50 |
| *** bhuvan has joined #openstack-infra | 16:51 | |
| notmyname | fungi: which is another thing that's weird to me, since it means that the tests are run with requirements that may be different than are specified (as is the case here?) | 16:51 |
| notmyname | jeblair: but yeah, it seems there's some gap/bug here | 16:51 |
| *** gyee has quit IRC | 16:51 | |
| fungi | notmyname: which is pretty much always an expectation. problem being that different projects can possibly specify slightly different versions of requirements because they don't all get force-upgraded and are instead allowed to decide when they sync requirements versions | 16:52 |
| clarkb | notmyname: we have to do it that way otherwise we install python-keystoneclient 20 times | 16:52 |
| *** doude has quit IRC | 16:52 | |
| fungi | so we use global-reqs as the reference list of which versions to test with in integration test jobs do deal with the discrepancies between declared requirements in different projects (as they're nondeterministic if you just throw them all at the wall as listed) | 16:53 |
| *** laxity has quit IRC | 16:53 | |
| fungi | this goes back to the "python dependency resolution is serial, not holistic" problem | 16:54 |
| *** laxity has joined #openstack-infra | 16:54 | |
| notmyname | how do I `for each project in openstack; do grep "futures" requirements.txt; done` | 16:54 |
| notmyname | ie without either manually clicking on every project or having to clone every one | 16:55 |
| fungi | http://162.242.232.62:8080/search/livegrep perhaps if it's up to date | 16:56 |
| clarkb | notmyname: we don't have livegrep up yet so I think you ahve to clone each one. OH if there is another livegrep maybe use that | 16:56 |
| clarkb | notmyname: you can always shallow clone to make it go quick | 16:56 |
| fungi | clarkb: i think that one was just a one-time poc and may not be updated | 16:57 |
| *** doude has joined #openstack-infra | 16:57 | |
| fungi | also may not have had all the projects in it, but rather just some subset | 16:57 |
| * clayg didn't know clone had a --depth option | 16:57 | |
| fungi | clayg: yep, for making a shallow clone of arbitrary history length | 16:58 |
| *** ominakov has quit IRC | 16:58 | |
| clarkb | note not all git operations play nice with shallow cloned repos, but grep should be fine | 16:59 |
| clayg | notmyname: don't know what you're going after trying to get all the versions from the other projects? | 16:59 |
| Sincler | zaro: hi i'm having some troubles with the update of gerrit, about the changes in the database, i'm changing from 2.4.4 to 2.8.4 | 17:00 |
| *** gokrokve has joined #openstack-infra | 17:01 | |
| fungi | and to jeblair's earlier suggestion, http://logs.openstack.org/53/85453/30/check/check-grenade-dsvm/acc74de/logs/old/devstacklog.txt.gz#_2014-08-26_14_20_15_153 does indeed seem to have sync'd futures>=2.1.3 but that was the version in the stable/icehouse branch of openstack/requirements until | 17:02 |
| fungi | http://git.openstack.org/cgit/openstack/requirements/commit/global-requirements.txt?h=stable/icehouse&id=c99cbdc7d3a1e5f5b29b9f1507fad2e62e09485c | 17:02 |
| *** jp_at_hp has quit IRC | 17:02 | |
| *** dizquierdo has quit IRC | 17:02 | |
| jeblair | yeah, havana's the branch that broke, right? | 17:02 |
| Sincler | zaro can you help me? | 17:02 |
| fungi | oh, right... | 17:03 |
| *** doude has quit IRC | 17:03 | |
| *** annegent_ has joined #openstack-infra | 17:03 | |
| clayg | clarkb: so to avoid "install python-keystoneclient 20 times" - what do we do? just install what ever is in global-requirements at the top, and then when you install all the projects and clients, every depends says something like "blah>=2.1.3 requirement already satisfied" ??? | 17:03 |
| clarkb | jeblair: I am catching up on scrollback. It looks like we are going to reinstate fresh check? | 17:03 |
| *** harlowja_away is now known as harlowja_ | 17:03 | |
| jeblair | clarkb: i would not go that far | 17:03 |
| jeblair | clarkb: we have found 2 changes that it would have caught | 17:03 |
| *** pdmars has quit IRC | 17:04 | |
| *** harlowja_ is now known as harlowja_away | 17:04 | |
| clarkb | clayg: we ensure all projects match global requirements then the first project for each dependency installs that dependency and since all requirements are the same it is satisfied for everyone | 17:04 |
| jeblair | clarkb: i proposed the idea that if we decide it's worth it, perhaps we should reconsider a different value for the time cutoff | 17:04 |
| clarkb | clayg: so basically yes. | 17:04 |
| jeblair | clarkb, fungi: i am going to go into a hole and debug nodepool for a while | 17:04 |
| clarkb | jeblair: ok and thanks | 17:04 |
| jeblair | anything you need before i do so? i'll check back in in a few hours | 17:05 |
| clarkb | nothing from me. still catching up on morning | 17:05 |
| clarkb | email and bugs and irc | 17:05 |
| fungi | jeblair: notmyname: yeah, so the reason grenade didn't catch that for the swiftclient change is that swiftclient has only a master branch and so only has an icehouse->master grenade job (no havana->icehouse). i'm looking at the backward-compat havana tempest job now to see if it would have made any difference there | 17:05 |
| *** pballand has quit IRC | 17:05 | |
| phschwartz | fungi: clearing the instances now and watching for new ones that fail in dfw | 17:05 |
| fungi | jeblair: sounds good | 17:05 |
| fungi | phschwartz: thanks! looks like we're showing 152 in error+deleting in dfw at the moment | 17:06 |
| fungi | i'll watch for that to drop | 17:06 |
| clayg | clarkb: but not all projects match global requirements | 17:07 |
| clarkb | clayg: right, we force them to in the integration tests | 17:07 |
| *** alkari has quit IRC | 17:08 | |
| notmyname | fungi: isn't that the same for all client libs? just a master? | 17:08 |
| notmyname | clayg: I was wonder what others use to figure out if it would be possible to use 2.1.2 rather than 2.1.3 | 17:08 |
| *** pballand has joined #openstack-infra | 17:08 | |
| clarkb | there is a requirements merge step for each project | 17:08 |
| fungi | clarkb: by in-place editing their requirements lists to have version strings matching openstack/requirements:global-requirements.txt | 17:08 |
| fungi | during the job set-up | 17:08 |
| clarkb | notmyname: my guess is it is ok. And if you propose that global requirements update there is a reasonable amouint of coverage for it | 17:08 |
| fungi | s/clarkb/clayg/ | 17:08 |
| clarkb | notmyname: which should give us a quick smoke test then we can work on the wider picture if we like | 17:09 |
| *** harlowja_away is now known as harlowja_ | 17:09 | |
| *** mpaolino has quit IRC | 17:09 | |
| notmyname | clarkb: since this is something that we added to a client lib, I was hoping for using the version in precise, just to lower pain if someone is packaging it. they could use the distro packages | 17:09 |
| *** andreaf_ has quit IRC | 17:09 | |
| *** andreaf_ has joined #openstack-infra | 17:10 | |
| *** e0ne has joined #openstack-infra | 17:10 | |
| *** denis_makogon has quit IRC | 17:10 | |
| clarkb | notmyname: ya its worth doing for that reason | 17:12 |
| *** tonytan4ever has quit IRC | 17:12 | |
| clarkb | I am always wary of rolling back min versions though as we have a tendency to bump those to minimum versions that were known to work at one point | 17:12 |
| *** pblaho has quit IRC | 17:13 | |
| fungi | huh... so i don't see futures being addressed at all in http://logs.openstack.org/53/85453/30/check/check-tempest-dsvm-full-havana/78a1a39/logs/devstacklog.txt.gz#_2014-08-26_14_31_50_657 | 17:13 |
| *** tonytan4ever has joined #openstack-infra | 17:13 | |
| fungi | i wondn't be surprised if the sync operation there is ignoring requirements which are specified in the project but not in the global requirements list? | 17:13 |
| clarkb | fungi: maybe it is a new dependency? | 17:13 |
| openstackgerrit | John Dickinson proposed a change to openstack/requirements: lower the futures version to 2.1.2 https://review.openstack.org/117569 | 17:13 |
| notmyname | clayg: ^^ | 17:14 |
| *** sdake_ has joined #openstack-infra | 17:14 | |
| notmyname | er. clarkb^^ | 17:14 |
| fungi | clarkb: this is the havana backward-compat test for the swiftclient change adding it | 17:14 |
| fungi | clarkb: and that requirement isn't in the havana branch of requirements, so isn't addressed by the sync at all looks like | 17:14 |
| *** annegent_ has quit IRC | 17:14 | |
| clarkb | fungi: that would do it | 17:15 |
| *** trinaths has quit IRC | 17:15 | |
| fungi | basically trying to suss out whether we want to fail devstack-based jobs where there are requirements not listed in global-reqs | 17:15 |
| fungi | mtreinish: ^ didn't you have a change at one point which was checking that? | 17:16 |
| fungi | can't remember if that was something else, or if it isn't happening on havana devstack maybe, or if we reverted it for some reason | 17:16 |
| clayg | what decides if a check-grenade-dsvm job checks out stable/icehouse or stable/havana when it's setting up "old" | 17:16 |
| *** RicoLin has quit IRC | 17:17 | |
| *** dane_leblanc has quit IRC | 17:17 | |
| *** markmcclain has quit IRC | 17:17 | |
| clarkb | clayg: the new branch which is determined by the change being tested. So if the change tested is for icehouse then old will be havana. If the change tested is master/juno old will be icehouse | 17:17 |
| clarkb | clayg: because we only test n-1 to n | 17:18 |
| Sincler | zaro: can you help me? I'm updating gerrit from 2.4.4 to 2.8.4, I noticed that some tables are droped like, account_agreements, approval_categories, contributor_agreements, do create te tables , you are using .config files or what you have done? | 17:18 |
| clayg | clarkb: ok, that makes sense - but I don't get why the new python-swiftclient change was tested against havana then | 17:19 |
| *** jlibosva has quit IRC | 17:19 | |
| clarkb | clayg: because clients have no stable branches we test master clients against old stable releases of servers | 17:19 |
| clayg | clarkb: fungi: this is job that notmyname linked me this morning that failed with the swiftclient importerror -> http://logs.openstack.org/91/116991/1/check/check-grenade-dsvm/682b033/logs/grenade.sh.txt.gz#_2014-08-26_22_51_00_697 | 17:19 |
| *** dustins has joined #openstack-infra | 17:20 | |
| *** dguerri is now known as dguerri`afk | 17:20 | |
| *** Ryan_Lane has joined #openstack-infra | 17:20 | |
| clayg | clarkb: oh how interesting, but we don't install their dependencies - that can be tricky | 17:20 |
| clarkb | we do install their dependencies... | 17:20 |
| *** praneshp has joined #openstack-infra | 17:21 | |
| clarkb | I think what fungi discovered was that the merge from global requirements is only for things both sides know about and that may have broken this? | 17:21 |
| clayg | i may have overgeneralized - in this case the dependencies specified in the new version of the client weren't installed -> http://logs.openstack.org/91/116991/1/check/check-grenade-dsvm/682b033/logs/grenade.sh.txt.gz#_2014-08-26_23_06_34_640 | 17:21 |
| clarkb | fungi: ^ | 17:21 |
| fungi | clayg: the assertion is that clients will work with old versions of servers, so we try to test them with all the server integrated releases for which we can currently support testing (right now havana and icehouse) | 17:21 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: the help files changed from _ to - at some point https://review.openstack.org/117572 | 17:21 |
| *** sandywalsh has quit IRC | 17:21 | |
| dhellmann | clarkb: how did anything get into the project requirements list if it wasn't in the global requirements list? | 17:22 |
| fungi | clarkb: yeah, i have a suspicion that is the case and i'm looking back through that function now | 17:22 |
| clarkb | dhellmann: ~havana time frame you could do that iirc | 17:22 |
| fungi | dhellmann: it was in new versions of the requirements list (icehouse and master) but not the havana branch of the requirements list | 17:22 |
| * dhellmann doesn't even remember last week | 17:22 | |
| *** sdake_ has quit IRC | 17:22 | |
| *** zehicle_at_dell has quit IRC | 17:22 | |
| dhellmann | got it, I didn't realize this was looking so far back | 17:22 |
| *** jamesmcarthur has joined #openstack-infra | 17:23 | |
| fungi | dhellmann: and was a change to a client, so only master branch there | 17:23 |
| *** paulrad has quit IRC | 17:23 | |
| jamesmcarthur | fungi: good afternoon! | 17:23 |
| fungi | dhellmann: so the requirements check passed it because master client branch master requirements branch, all good | 17:23 |
| jamesmcarthur | fungi: have you guys had any problem with the endpoint? | 17:23 |
| dhellmann | fungi: right, we see new requirements that have to be added to the stable branches fairly often | 17:23 |
| fungi | jamesmcarthur: none reported so far, no | 17:23 |
| jamesmcarthur | fungi: great | 17:23 |
| mtreinish | fungi: I don't think it was me | 17:24 |
| mtreinish | but it might have been... | 17:24 |
| *** ociuhandu has quit IRC | 17:24 | |
| fungi | mtreinish: i was probably thinking of one of your dopplegangers then | 17:24 |
| jamesmcarthur | fungi: so we’re ready to flip the switch on cloudflare as soon as you guys are. What’s the turn around from your side to chagne gerrit to look at direct.openstack.org i/o www.openstack.org? | 17:24 |
| fungi | jamesmcarthur: excellent work btw, i didn't even see a blip | 17:24 |
| jamesmcarthur | fungi: so your test would be curl --data "APPSEC=yEKVvoxEdGPxevE2&email=todd@openstack.org" http://direct.openstack.org/verify/member | 17:24 |
| jamesmcarthur | fungi: thanks! yeah, it was pretty smooth. very pleased :) | 17:25 |
| clarkb | fungi: jamesmcarthur: that requires a gerrit restart right? maybe we can do that on saturday? | 17:25 |
| *** dane_leblanc has joined #openstack-infra | 17:25 | |
| dhellmann | clarkb, SergeyLukjanov: do either of you have time to look at this oslo.log import request today? I'd like to move ahead with that so we can start cleaning up the incubator. http://review.openstack.org/#/c/116934/ | 17:25 |
| fungi | clarkb: jamesmcarthur: yeah, we can test it manually, but can't switch it in production until this weekend | 17:25 |
| jamesmcarthur | clarkb: fungi: ok, should I put in a launchpad ticket for it? | 17:25 |
| *** SumitNaiksatam has quit IRC | 17:25 | |
| fungi | jamesmcarthur: nah, i'll just go ahead and write up the change | 17:26 |
| clarkb | dhellmann: done | 17:26 |
| fungi | jamesmcarthur: and we'll lump it in with a maintenance we already have scheduled for 12:00 utc on saturday | 17:26 |
| clarkb | 1200 utc are you mad? | 17:26 |
| clarkb | I thought it was 1600 utc? | 17:26 |
| jamesmcarthur | fungi: awesome! thanks sir! I’ll let the Foundation know and we’ll plan on turning CloudFlare on after we hear from you. | 17:26 |
| clarkb | 5am PDT is way too early | 17:26 |
| *** alkari has joined #openstack-infra | 17:27 | |
| jamesmcarthur | clarkb: come on! you don’t like to get up at 5am on Saturday ;) | 17:27 |
| *** paulrad has joined #openstack-infra | 17:27 | |
| clarkb | jamesmcarthur: only when fishing and even then may not be worth it | 17:27 |
| fungi | er, yeah, 1600utc, oops ;) | 17:27 |
| *** jerryz has joined #openstack-infra | 17:28 | |
| dhellmann | clarkb: thanks! | 17:29 |
| *** HeOS has quit IRC | 17:29 | |
| jamesmcarthur | clarkb: fungi: thanks guys for your help. I’m glad everyone gets to sleep in a bit. | 17:29 |
| *** _nadya_ has joined #openstack-infra | 17:29 | |
| *** jamesmcarthur has left #openstack-infra | 17:30 | |
| fungi | jamesmcarthur: and i tested that curl manually from our code review server, no problem | 17:30 |
| fungi | oh, he already left | 17:30 |
| *** paulrad has quit IRC | 17:31 | |
| *** dane_leblanc has quit IRC | 17:32 | |
| jerryz | fungi: the py27,p26 tests for our stackforge project failed because a package was not found on pypi.o.o, http://logs.openstack.org/34/117434/1/check/gate-compass-core-python26/600a4e3/console.html | 17:32 |
| clarkb | oh transifex, everytime you send me a couple hundred emails all at once I want pootle/zanata already | 17:32 |
| *** mmedvede has quit IRC | 17:33 | |
| *** mmedvede has joined #openstack-infra | 17:33 | |
| *** dkranz is now known as dkranz_afk | 17:33 | |
| *** bdpayne has joined #openstack-infra | 17:34 | |
| clarkb | jerryz: the package name you used is wrong | 17:34 |
| clarkb | It should be Flask-WTF not flask-wtf | 17:34 |
| clarkb | if pip has to change the name for you like that it apparently treats it as an external packagel ink | 17:34 |
| clarkb | dstufft said he wanted to fix that in the nearish future | 17:34 |
| *** tonytan4ever has quit IRC | 17:34 | |
| fungi | clarkb: though in this case it looks like it might be an index/sync issue? http://pypi.openstack.org/simple/Flask-WTF/ has a link for Flask_WTF-0.10.1-py2.py3-none-any.whl which returns 404 | 17:35 |
| *** dane_leblanc has joined #openstack-infra | 17:35 | |
| clarkb | oh its this bug again | 17:35 |
| jerryz | clarkb: but it got redirected to http://pypi.openstack.org/packages/2.7/F/Flask-WTF/ | 17:36 |
| fungi | the wheel was uploaded 2014-08-26 though | 17:36 |
| clarkb | jerryz: yes, it will redirect but then pip will say that is an external package link and fail | 17:36 |
| fungi | two days ago, give or take | 17:36 |
| clarkb | jerryz: so the redirect doesn't help | 17:36 |
| *** dangers is now known as dangers_away | 17:36 | |
| *** daya_k has joined #openstack-infra | 17:36 | |
| *** andreaf_ has quit IRC | 17:36 | |
| *** sabeen has quit IRC | 17:36 | |
| *** zehicle_at_dell has joined #openstack-infra | 17:36 | |
| fungi | yeah, we're still hitting 2014-08-28 17:35:04,020 ERROR: Error syncing package: instant-rst@1204675 | 17:36 |
| clarkb | that will do it | 17:37 |
| *** zul has quit IRC | 17:37 | |
| *** sdake_ has joined #openstack-infra | 17:37 | |
| *** andreaf has joined #openstack-infra | 17:37 | |
| *** dkranz_afk is now known as dkranz | 17:38 | |
| *** SumitNaiksatam has joined #openstack-infra | 17:38 | |
| jerryz | clarkb: so for the moment, we need change all our requirements' names to match the letter case on the pypi.o.o server? | 17:38 |
| clarkb | jerryz: there is a bug in bandersnatch where it can sync the index page and not all of the actual packages | 17:38 |
| clarkb | jerryz: we are hitting that which caused this specific issue | 17:38 |
| clarkb | jerryz: once that is fixed I think the case issue will break you too | 17:39 |
| *** dkehn_ has joined #openstack-infra | 17:39 | |
| clarkb | jerryz: you don't have to | 17:39 |
| clarkb | but you have to allow external package links | 17:39 |
| clarkb | I think making the case correct is a better choice | 17:39 |
| *** Sukhdev has joined #openstack-infra | 17:40 | |
| *** pcm_ has quit IRC | 17:40 | |
| fungi | dstufft: we're hitting a fastly redirect loop because instant_rst got renamed to instant-rst and bandersnatch hasn't been able to proceed past that serial for a few days now as a result. if you get a chance to blow that away, it would be appreciated | 17:40 |
| *** pcm_ has joined #openstack-infra | 17:41 | |
| *** dkehn_ has quit IRC | 17:41 | |
| *** dkehn_ has joined #openstack-infra | 17:41 | |
| dstufft | fungi: should be fixed now | 17:41 |
| *** dkehn has quit IRC | 17:42 | |
| *** dkehnx has quit IRC | 17:42 | |
| *** markmcclain has joined #openstack-infra | 17:42 | |
| *** hogepodge has joined #openstack-infra | 17:43 | |
| *** alkari has quit IRC | 17:43 | |
| *** dkehn has joined #openstack-infra | 17:43 | |
| *** rfolco has quit IRC | 17:43 | |
| fungi | dstufft: thanks! i'll keep an eye on it | 17:45 |
| *** alkari has joined #openstack-infra | 17:45 | |
| *** yolanda has quit IRC | 17:47 | |
| *** andreaf has quit IRC | 17:47 | |
| *** andreaf has joined #openstack-infra | 17:47 | |
| *** yfried_ has joined #openstack-infra | 17:48 | |
| *** eharney has joined #openstack-infra | 17:48 | |
| *** zul has joined #openstack-infra | 17:50 | |
| *** amcrn has joined #openstack-infra | 17:51 | |
| *** emagana has quit IRC | 17:51 | |
| *** emagana has joined #openstack-infra | 17:51 | |
| *** andreaf has quit IRC | 17:51 | |
| *** Ajaeger has joined #openstack-infra | 17:51 | |
| *** marun has quit IRC | 17:52 | |
| *** amuller_ has quit IRC | 17:52 | |
| *** dtantsur is now known as dtantsur|afk | 17:52 | |
| jerryz | fungi: how soon should bandersnatch be running again to sync the packeges | 17:52 |
| jeblair | i just popped in to say i'm waiting on my fake nodepool to build fake images and it's taking a while. :) | 17:52 |
| clarkb | jerryz: its ~every 5 minutes once something like that is fixed | 17:53 |
| clarkb | jerryz: you can always check the mirror directly | 17:53 |
| jeblair | it had to build 67 of them | 17:53 |
| *** yolanda has joined #openstack-infra | 17:54 | |
| *** andreaf has joined #openstack-infra | 17:54 | |
| *** emagana_ has joined #openstack-infra | 17:55 | |
| *** emagana has quit IRC | 17:55 | |
| *** rfolco has joined #openstack-infra | 17:55 | |
| clarkb | jerryz: fungi looks like that wheel hasn't synced yet | 17:55 |
| *** emagana_ has quit IRC | 17:56 | |
| *** _nadya_ has quit IRC | 17:57 | |
| *** morganfainberg is now known as morganfainberg_Z | 17:57 | |
| *** sdake_ has quit IRC | 17:57 | |
| clarkb | I was able to get it from pypi.python.org/simple/Flask-WTF so it does exist | 17:57 |
| *** mfer has quit IRC | 17:58 | |
| fungi | yeah, it generally takes a couple of subsequent bandersnatch runs to catch back up | 17:58 |
| fungi | usually the first run just completes the one transaction it was previously unable to perform, then the next run does all the pending transactions after it to present (if possible) | 17:59 |
| *** reed has joined #openstack-infra | 17:59 | |
| *** pelix1 has quit IRC | 17:59 | |
| fungi | i'm watching to see if the 1800 run in a moment picks it up | 17:59 |
| *** johnthetubaguy is now known as zz_johnthetubagu | 17:59 | |
| *** tonytan4ever has joined #openstack-infra | 18:00 | |
| *** mmaglana has quit IRC | 18:01 | |
| *** tonytan4ever has quit IRC | 18:01 | |
| *** tonytan4ever has joined #openstack-infra | 18:02 | |
| fungi | yeah, it caught the rest of the way up, but somehow never added a Flask-WTF directory under http://pypi.openstack.org/packages/2.7/F/ | 18:02 |
| *** annegent_ has joined #openstack-infra | 18:03 | |
| *** fifieldt has joined #openstack-infra | 18:04 | |
| *** _nadya_ has joined #openstack-infra | 18:06 | |
| clarkb | maybe we need to lgiht it on fire and have it regrab? | 18:06 |
| *** tonytan4ever has quit IRC | 18:06 | |
| *** armax has quit IRC | 18:06 | |
| *** annegent_ has quit IRC | 18:07 | |
| *** pcm_ has quit IRC | 18:08 | |
| *** mestery has quit IRC | 18:08 | |
| *** mestery has joined #openstack-infra | 18:08 | |
| *** pcm_ has joined #openstack-infra | 18:09 | |
| *** gondoi is now known as zz_gondoi | 18:09 | |
| fungi | by light it on fire you mean reset its serial to some old value? | 18:09 |
| *** bradjones has quit IRC | 18:09 | |
| *** armax has joined #openstack-infra | 18:11 | |
| *** zz_gondoi is now known as gondoi | 18:11 | |
| *** sdake has quit IRC | 18:12 | |
| *** andreaf has quit IRC | 18:15 | |
| *** mwagner_lap has joined #openstack-infra | 18:18 | |
| *** andreaf has joined #openstack-infra | 18:19 | |
| *** paulrad has joined #openstack-infra | 18:19 | |
| *** annegent_ has joined #openstack-infra | 18:19 | |
| *** armax has quit IRC | 18:21 | |
| *** armax has joined #openstack-infra | 18:21 | |
| *** gyee has joined #openstack-infra | 18:21 | |
| clarkb | fungi: I was thinking deleting flask-wtf's index page | 18:22 |
| *** pballand has quit IRC | 18:22 | |
| clarkb | but I don't actually know enough about bandersnatch to know if that helps | 18:22 |
| clarkb | dstufft: ^ | 18:22 |
| dstufft | I don't think bandersnatch works by computing deltas | 18:23 |
| dstufft | it uses a journal | 18:23 |
| fungi | clarkb: yeah, i got the impression bandersnatch only rolls forward based on modifications indicated between serial points, so unless we replayed the point which updated it... | 18:23 |
| fungi | and presumably we could roll back and replay from an old enough point in the journal, but not sure how to safely rewind it nor whether that would cause problems with deletions which have already happened, et cetera | 18:24 |
| dstufft | you can make it do a full sync though | 18:25 |
| dstufft | https://pypi.python.org/pypi/bandersnatch#maintenance | 18:25 |
| *** lcheng_ has joined #openstack-infra | 18:26 | |
| fungi | so in theory just deleting the state and todo files and letting it rerun on the next pulse... we could do that easily enough though that might impact freshness for a good chunk of today | 18:27 |
| fungi | clarkb: opinion? | 18:27 |
| *** salv-orlando has joined #openstack-infra | 18:28 | |
| *** andreaf has quit IRC | 18:29 | |
| clarkb | fungi: if that is the prescribed fix I suppose we should giev it a go | 18:29 |
| clarkb | fungi: bit worried htat other things may be similarly broken and running it this way will correct that | 18:30 |
| fungi | okay, doing now. the earlier we get it going the sooner it will be back up to date | 18:30 |
| *** dustins has quit IRC | 18:30 | |
| *** dkehn_ is now known as dkehnx | 18:31 | |
| *** praneshp has quit IRC | 18:32 | |
| *** annegent_ has quit IRC | 18:33 | |
| *** pcm_ has quit IRC | 18:33 | |
| clarkb | how do people get any work done with all of these mega ml threads | 18:33 |
| *** annegent_ has joined #openstack-infra | 18:33 | |
| *** pcm_ has joined #openstack-infra | 18:33 | |
| dstufft | bandersnatch is at least smartish about a full sync | 18:34 |
| dstufft | it won't redownload files that it already has which hashes match | 18:34 |
| *** mrmartin has joined #openstack-infra | 18:35 | |
| dstufft | so a full sync really is a delta, but at the cost of a uh, at least ~90k HTTP requests | 18:35 |
| dstufft | at least | 18:35 |
| *** praneshp has joined #openstack-infra | 18:35 | |
| dstufft | more to actually fic errors | 18:35 |
| fungi | yeah, our dos attack against pypi.python.org (well fastly's cache anyway) is under way now | 18:36 |
| openstackgerrit | Joshua Harlow proposed a change to openstack-infra/config: Switch taskflow to use the unittests slave script https://review.openstack.org/116388 | 18:37 |
| jeblair | DEBUG:nodepool.NodePool: <AllocationGrant of 1370 of devstack-trusty from fake-provider7> | 18:38 |
| jeblair | progress! ^ | 18:38 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Make list of open reviews bold https://review.openstack.org/117588 | 18:38 |
| clarkb | jeblair: nice | 18:38 |
| jogo | fungi: I added the bold tag ^ I skipped the blinking one like you suggested | 18:38 |
| clarkb | jogo: what about marquee? | 18:38 |
| fungi | i think i actually suggested flaming text and spinning skulls, but close enough! | 18:39 |
| jeblair | jogo: if you're making changes to that page, would you please add an animated under construction icon? ;) | 18:39 |
| fungi | jeblair: so you managed to get it to grant 1370 with a max_servers in fake-provider7 of how many? | 18:40 |
| jeblair | perhaps one of these: http://www.textfiles.com/underconstruction/ | 18:40 |
| jeblair | fungi: 55 | 18:40 |
| *** sdake_ has joined #openstack-infra | 18:41 | |
| fungi | wowza | 18:41 |
| *** dustins has joined #openstack-infra | 18:41 | |
| *** sdake_ has quit IRC | 18:42 | |
| *** sdake has joined #openstack-infra | 18:42 | |
| *** sdake has quit IRC | 18:42 | |
| *** sdake has joined #openstack-infra | 18:42 | |
| clarkb | morganfainberg_Z: for 116404 I thought you had handled keystone logs in d-g | 18:42 |
| clarkb | morganfainberg_Z: want ot make sure that we aren't duplicating work somewhere | 18:42 |
| *** armax has quit IRC | 18:43 | |
| *** armax has joined #openstack-infra | 18:43 | |
| *** todd_dsm has joined #openstack-infra | 18:44 | |
| *** todd_dsm has quit IRC | 18:44 | |
| clarkb | salv-orlando: for 116129 aren't extensions queryable via the apis? | 18:44 |
| clarkb | salv-orlando: is that change made in order to control which ones are enabled or in order to notify tempest of which are enabled? | 18:45 |
| *** e0ne has quit IRC | 18:46 | |
| clarkb | sdague: should 100229 be WIP until you have fixed the pid stuff? | 18:46 |
| *** hashar has joined #openstack-infra | 18:47 | |
| *** e0ne has joined #openstack-infra | 18:50 | |
| adam_g | clarkb, one more quick fix for the sideways stuff https://review.openstack.org/#/c/117435/ | 18:52 |
| clarkb | gah my bash is bad and I should feel bad | 18:53 |
| *** _nadya_ has quit IRC | 18:54 | |
| clarkb | +2 | 18:55 |
| *** dustins has quit IRC | 18:55 | |
| clarkb | fungi: jeblair ^ was a silly derp on my part that adam_g is fixing may be good to review soonish | 18:55 |
| *** sdake has quit IRC | 18:56 | |
| *** daya_k has quit IRC | 18:56 | |
| adam_g | thanks | 18:57 |
| fungi | clarkb: looks like it passed all jobs except for the non-voting f20 one | 18:57 |
| fungi | lgtm too | 18:57 |
| fungi | once it eventually gets an f20 node for a few minutes, it should head straight on into the gate | 18:57 |
| fungi | looks like there are 14 changes ahead of it in check also waiting for f20 nodes | 18:58 |
| clarkb | ok I am cuaght up on infra-specs reviews for the important specs | 18:58 |
| *** tonytan4ever has joined #openstack-infra | 18:58 | |
| openstackgerrit | A change was merged to openstack-infra/infra-specs: Remove docutils pin https://review.openstack.org/117169 | 18:59 |
| *** ominakov has joined #openstack-infra | 18:59 | |
| *** gondoi is now known as zz_gondoi | 19:00 | |
| *** amotoki_ has joined #openstack-infra | 19:02 | |
| *** dustins has joined #openstack-infra | 19:02 | |
| *** masayukig has quit IRC | 19:02 | |
| *** sdake has joined #openstack-infra | 19:02 | |
| *** sdake has quit IRC | 19:02 | |
| *** sdake has joined #openstack-infra | 19:02 | |
| *** amotoki has quit IRC | 19:03 | |
| Ajaeger | clarkb, fungi: Could you put on your review queue the patches by dhellmann, please? https://review.openstack.org/#/c/117337/ and the dependencies | 19:03 |
| *** cnesa8 has quit IRC | 19:05 | |
| *** zz_gondoi is now known as gondoi | 19:05 | |
| *** lcheng_ has quit IRC | 19:06 | |
| jeblair | clarkb, fungi, ianw: i have a solidly reproducible test procedure now. it's huge because it nearly completely mimics what we saw in production. | 19:08 |
| jeblair | i'm going to try to whittle it down now | 19:08 |
| *** mfer has joined #openstack-infra | 19:08 | |
| jeblair | well, after lunch | 19:08 |
| clarkb | jeblair: that is great news | 19:08 |
| clarkb | oh I shoudl lunch too | 19:08 |
| clarkb | jeblair: the cloner change for cache dir lgtm | 19:08 |
| fungi | jeblair: that's thrilling. i'm excited to see the distillation | 19:08 |
| *** masayukig has joined #openstack-infra | 19:08 | |
| *** fifieldt has quit IRC | 19:08 | |
| *** sdake has quit IRC | 19:08 | |
| *** sdake has joined #openstack-infra | 19:09 | |
| *** sdake has quit IRC | 19:09 | |
| *** sdake has joined #openstack-infra | 19:09 | |
| jeblair | clarkb: check out the status page | 19:11 |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/config: Add list of types of logs to tempest-logs page https://review.openstack.org/117390 | 19:11 |
| *** yolanda has quit IRC | 19:11 | |
| mtreinish | jogo: ^^^ first set of updates | 19:11 |
| jeblair | clarkb: it looks like a currently running change is marked 'grey', probably because the window shrunk after it had started | 19:11 |
| jeblair | clarkb: i think that's a minor display bug, we should probably not mark a change as 'grey' after it's actually started :) | 19:12 |
| clarkb | jeblair: I think that is correct. And it had to do with not wanting to throw away jobs we already started | 19:12 |
| clarkb | jeblair: so ya the ui can display that better | 19:12 |
| jogo | mtreinish: sweet | 19:13 |
| *** chuck_ has joined #openstack-infra | 19:14 | |
| adam_g | do slaves default to distro packages of python libraries if the package version available satisfies requirements.txt? | 19:14 |
| adam_g | (devstack, specifically) | 19:14 |
| Ajaeger | clarkb: Thank you! | 19:15 |
| *** balles has joined #openstack-infra | 19:15 | |
| clarkb | Ajaeger: see comments on the doc8 changes | 19:16 |
| jogo | mtreinish: why did you remove heat? | 19:16 |
| fungi | adam_g: nope | 19:16 |
| *** zul has quit IRC | 19:16 | |
| jogo | mtreinish: and thanks again, it looks good | 19:16 |
| Ajaeger | clarkb: yeah, using doc sounds better. Let's see what dhellmann thinks... | 19:17 |
| fungi | adam_g: or for a longer answer, "it's complicated" (depends on whether you're talking about how we set up devstack vs things we run from python virtualenvs via tox) | 19:17 |
| * dhellmann perks up his ears | 19:17 | |
| mtreinish | jogo: did I? oops | 19:17 |
| salv-orlando | clarkb: change 116129 is for preparing devstack so that it will notify tempest about the right extensions to test. Extensions can be queried, but that is not possible from devstack gate as services are not yet running. | 19:17 |
| adam_g | fungi, trying to figure out how we end up /w eventlet==0.13.0 while requirements.txt lists eventlet>=0.13.0 and pypi has 0.15.1 | 19:17 |
| mtreinish | jogo: I've got some more I'm adding right now, I'll add heat back as part of that rev | 19:17 |
| Ajaeger | clarkb: we can easily gate on doc8 if you like... | 19:17 |
| adam_g | trusty ships 0.13.0 | 19:17 |
| fungi | adam_g: for things like unit tests, anything which is a requirement or a test requirement and then listed in the requirements.txt or test-requirements.txt of a project as such gets installed into a virtualenv by tox (latest versions on pypi matching the stated version specs in those files) | 19:18 |
| clarkb | dhellmann: Ajaeger I don't really care what the actual target is, just pointing out we are inconsistent in places and should sort that out (that can be a different change) | 19:18 |
| *** balles has quit IRC | 19:18 | |
| adam_g | fungi, im talking about what is actually used by devstack services | 19:18 |
| *** sballe has joined #openstack-infra | 19:18 | |
| jogo | mtreinish: awesome | 19:18 |
| dhellmann | clarkb: the tox env name? yeah, I don't care one way or the other. Did you count what we're doing now by any chance? | 19:18 |
| clarkb | salv-orlando: they are running when tempest is running though | 19:18 |
| clarkb | dhellmann: I didn't count. I have a hunch docs is more common | 19:18 |
| fungi | adam_g: okay, so devstack has lists of rpms and debs it installs dependeing on platform and what's enabled | 19:19 |
| adam_g | fungi, i see python-eventlet distro package being downloaded and installed as part of devstacks package install | 19:19 |
| fungi | adam_g: and devstack is running the servers in the global system context rather than in a virtualenv | 19:19 |
| Ajaeger | dhellmann: I think most other projects put doc8 into the docs environment - but there are few projects using doc8 yet | 19:20 |
| dhellmann | clarkb: I see only "docs" in projects under openstack/* | 19:20 |
| clarkb | dhellmann: ok, I think doc started in zuul and JJB | 19:20 |
| clarkb | dhellmann: so we can go with docs and change the infra repos | 19:20 |
| dhellmann | oh, you want to join doc8 and docs? that's fine, too | 19:20 |
| *** chuck_ has quit IRC | 19:20 | |
| openstackgerrit | A change was merged to openstack-infra/config: Create repo for oslo.log https://review.openstack.org/116934 | 19:20 |
| dhellmann | \o/ | 19:20 |
| fungi | adam_g: yep, http://git.openstack.org/cgit/openstack-dev/devstack/tree/files/apts/nova#n38 | 19:20 |
| *** zul has joined #openstack-infra | 19:20 | |
| clarkb | dhellmann: no I don't want to join them. I don't know what doc8 gives us | 19:20 |
| adam_g | fungi, right, so we're dependent on distro packages for that | 19:20 |
| clarkb | dhellmann: I am proxying jeblair right now | 19:20 |
| fungi | adam_g: yes, which is why i said "it depends" ;) | 19:21 |
| Ajaeger | dhellmann: I see doc8 gating only in the documentation projects, so that's anecdotal reference ;) | 19:21 |
| clarkb | adam_g: fungi: is eventlet not listed as a dependency in nova? | 19:21 |
| clarkb | adam_g: fungi: beacuse imo eventlet is a thing that nova requires and should be installed via pip in devstack | 19:21 |
| dhellmann | clarkb: it's a linter. I don't tend to use it, but Ajaeger pointed out some things that it found so I thought I would submit the change after I'd added it locally. Not a big deal if you want to nix that | 19:21 |
| adam_g | clarkb, it is | 19:21 |
| clarkb | the things that devstack installs via apt or yum tend to be the libvirt/xen/mysql/postgres et al item | 19:21 |
| adam_g | it is also listed in devstack apts | 19:21 |
| fungi | clarkb: i believe it is, but as long as the ubuntu-provided package meets the version requirement for nova, pip won't pull down a newer version will it? | 19:21 |
| *** flaper87 is now known as flaper87|afk | 19:21 | |
| *** sandywalsh has joined #openstack-infra | 19:22 | |
| clarkb | fungi: oh maybe not | 19:22 |
| adam_g | my original question! | 19:22 |
| adam_g | :) | 19:22 |
| clarkb | it may require the --upgrade flag | 19:22 |
| fungi | yeah, unless devstack calls pip with -U | 19:22 |
| clarkb | dstufft: ^ | 19:22 |
| clarkb | ok I am going to go get food now | 19:22 |
| clarkb | back in a big | 19:22 |
| clarkb | *bit | 19:22 |
| adam_g | we (ironic) need to start using the most recent eventlet, im wondering if i just bump it in requirements.txt to >= $disto_version, it will install from pip | 19:22 |
| adam_g | er, > $distro_version | 19:23 |
| adam_g | which would be *much* eaiser than doing an ubuntu SRU | 19:23 |
| clarkb | adam_g: in that case I think upgrade is implicit but I could be wrong | 19:23 |
| clarkb | dstufft: would know | 19:23 |
| dstufft | as long as ubunuto correctly drops the egg-info or dist-info and it satisifies the requirements, pip won't upgrade it | 19:23 |
| dstufft | drops in* | 19:23 |
| *** sdake_ has joined #openstack-infra | 19:23 | |
| salv-orlando | clarkb: totally right. For the tempest patch (https://review.openstack.org/#/c/116121/) it is a valuable approach. | 19:23 |
| dstufft | it's unusual for a linux distro to not have the egg-info/dist-info in place | 19:23 |
| adam_g | okay | 19:24 |
| fungi | looking in http://git.openstack.org/cgit/openstack-dev/devstack/tree/functions-common it does not seem to call pip install with -U or --upgrade | 19:24 |
| dstufft | so most likely, if a linux provided thing is installed, and it matches the version spec, then pip won't upgrade (unless you use -U or --upgrade) | 19:24 |
| fungi | which explains the version you're seeing used by the job in that case | 19:24 |
| salv-orlando | clarkb: however from the devstack-gate side is much easier to specify extensions to exclude rather than extensions to include | 19:24 |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/config: Add list of types of logs to tempest-logs page https://review.openstack.org/117390 | 19:24 |
| mtreinish | jogo: ^^^ | 19:24 |
| *** nkinder has left #openstack-infra | 19:25 | |
| jogo | mtreinish: you should add a co-authored-by tag | 19:25 |
| mtreinish | jogo: oh, ok. one sec | 19:26 |
| adam_g | dstufft, thanks | 19:26 |
| dstufft | np | 19:26 |
| jogo | mtreinish: also a typo | 19:26 |
| jogo | around keystone | 19:26 |
| openstackgerrit | Salvatore Orlando proposed a change to openstack-infra/elastic-recheck: Add footprint for bug 1362528 https://review.openstack.org/117468 | 19:26 |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/config: Add list of types of logs to tempest-logs page https://review.openstack.org/117390 | 19:26 |
| uvirtbot` | Launchpad bug 1362528 in neutron "cirros starts with file system in read only mode" [Medium,New] https://launchpad.net/bugs/1362528 | 19:26 |
| *** e0ne has quit IRC | 19:27 | |
| adam_g | that begs the question that clarkb brought up, should devstack be listing python-eventlet in its list of required packages or should we be relying on requirements.txt for that instead? | 19:27 |
| *** joools has joined #openstack-infra | 19:27 | |
| clarkb | I like requirements.txt but lost thst battle with mysql | 19:28 |
| *** pdmars_ has joined #openstack-infra | 19:29 | |
| *** pdmars_ has quit IRC | 19:29 | |
| fungi | yeah, this is a perennial debate | 19:29 |
| *** e0ne has joined #openstack-infra | 19:29 | |
| *** pcm_ has quit IRC | 19:29 | |
| *** e0ne has quit IRC | 19:30 | |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/config: Add list of types of logs to tempest-logs page https://review.openstack.org/117390 | 19:30 |
| adam_g | it looks like theres a million python-* things listed in devstack/files/apts/ | 19:30 |
| fungi | the primary argument i've heard is that devstack tries to be closer to what you'd have on the target distro rather than installing everything it possibly can from pypi | 19:30 |
| adam_g | and by a million i mean 105 | 19:30 |
| *** pcm_ has joined #openstack-infra | 19:30 | |
| adam_g | fungi, makes sense | 19:30 |
| fungi | 105 is almost a million, so good enough | 19:30 |
| adam_g | just hoping i can work around having to wait a month or two for a SRU to the ubuntu package | 19:31 |
| *** openstackgerrit has quit IRC | 19:31 | |
| *** openstackgerrit has joined #openstack-infra | 19:32 | |
| *** mestery has quit IRC | 19:35 | |
| fungi | though i suspect much of that is merely historical... likely when devstack was first written you really could install just about all of the basic requirements for common openstack services from your distro, and then later when that stopped being the case select requirements were pulled in from pypi as a stop-gap | 19:35 |
| *** mestery has joined #openstack-infra | 19:36 | |
| *** sdake_ has quit IRC | 19:36 | |
| *** sdake_ has joined #openstack-infra | 19:38 | |
| *** sdake_ has joined #openstack-infra | 19:38 | |
| openstackgerrit | Adam Gandelman proposed a change to openstack/requirements: Bump eventlet >= 0.15.1 https://review.openstack.org/117601 | 19:39 |
| *** homeless has quit IRC | 19:39 | |
| *** Sukhdev has quit IRC | 19:40 | |
| *** homeless has joined #openstack-infra | 19:41 | |
| *** cnesa has quit IRC | 19:42 | |
| mtreinish | jeblair, fungi: I was looking at the ML thread about the neutron incubator stuff and I'm wondering if I'm going to have an issue with the tempest-lib work | 19:42 |
| mtreinish | because I'm generating merge commits from the tempest repo into the lib for a set of files | 19:42 |
| mtreinish | doing it piecewise because that's how we were planning to roll it out | 19:42 |
| fungi | mtreinish: not using filter-branch til filter it down to just the set of files you need in the tempest-lib repo? | 19:43 |
| *** joools has quit IRC | 19:44 | |
| clarkb | the problem is combining repos | 19:44 |
| clarkb | separating with filter branch is fine | 19:44 |
| clarkb | just dont try to combine them later | 19:44 |
| mtreinish | fungi: yeah I'm using filter branch to do it. That's what I meant by generating a merge commit | 19:45 |
| mtreinish | clarkb: ok cool | 19:45 |
| mtreinish | yeah it's only a one way move | 19:45 |
| mtreinish | it'll just be deleted from the tempest tree after it's migrated | 19:45 |
| *** HeOS has joined #openstack-infra | 19:45 | |
| *** dims has joined #openstack-infra | 19:46 | |
| * mtreinish is disappointed that tempest-lib won the name vote | 19:47 | |
| fungi | mtreinish: right, the concern with the neutron incubator discussion is that the main proposal had been "work on this stuff out of tree and then combine the repositories later at graduation" which does indeed get quite messy | 19:47 |
| mtreinish | fungi: ok cool, I just wanted to double check before I started scripting the tempest->tempest-lib migration to make it less manual | 19:48 |
| *** cnesa has joined #openstack-infra | 19:48 | |
| fungi | looks like our building count is higher than it should be... looking into why | 19:48 |
| fungi | i don't see any nodes stuck in a building state at least | 19:50 |
| openstackgerrit | K Jonathan Harker proposed a change to openstack-infra/config: Reduce graphite's stats retention policy https://review.openstack.org/115116 | 19:51 |
| openstackgerrit | Andreas Jaeger proposed a change to openstack/requirements: Remove docutils-pin https://review.openstack.org/117602 | 19:51 |
| *** mwagner_lap has quit IRC | 19:52 | |
| Ajaeger | jeblair, sdague: We can try removing the docutils-pin again - all projects using requirements repo should be fine now... | 19:52 |
| fungi | oh, though we do have ~100 which have been in a building state in rax-dfw for 40 minutes (none over 30 minutes in any other provider/region) | 19:52 |
| *** datsun180b has quit IRC | 19:52 | |
| fungi | looks like it probably coincides with when the hung delete nodes got cleared out | 19:53 |
| *** dims has quit IRC | 19:53 | |
| fungi | so we immediately requested ~150 new instances there and about 2/3 of them haven't completed yet | 19:53 |
| *** dprince has quit IRC | 19:53 | |
| *** e0ne has joined #openstack-infra | 19:53 | |
| *** juzuluag has quit IRC | 19:54 | |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/config: Add list of types of logs to tempest-logs page https://review.openstack.org/117390 | 19:57 |
| mtreinish | dhellmann: ^^^ I added the neutron files to the list | 19:57 |
| dhellmann | mtreinish: thanks! | 19:58 |
| fungi | back on the requirements sync and backward-compat devstack jobs issue, i can't find anywhere we actually fail/abort the job for a requirement being used which is not in the global requirements list. i guess it bears discussing whether that's something we want, or if we want to close that hole some other way (such as through additional requirements enforcement jobs) | 20:00 |
| *** baoli has quit IRC | 20:00 | |
| *** unicell has quit IRC | 20:01 | |
| *** e0ne has quit IRC | 20:03 | |
| sdague | fungi: so... we don't go critical, we just remove things from the list. | 20:03 |
| fungi | sdague: oh, not a bad idea | 20:03 |
| sdague | fungi: ok, I'm confused, that's not an idea, but a fact :) | 20:04 |
| *** pballand has joined #openstack-infra | 20:04 | |
| fungi | sdague: or are you saying update.py does that already? looking again, but it's not the behavior we saw in http://logs.openstack.org/53/85453/30/check/check-tempest-dsvm-full-havana/78a1a39/logs/devstacklog.txt.gz#_2014-08-26_14_31_50_657 | 20:04 |
| sdague | but I just got back to computer land in a new location, so I don't know if I have enough context | 20:04 |
| Ajaeger | fungi: see https://jenkins.openstack.org/job/propose-requirements-updates/186/consoleText | 20:05 |
| *** pcm_ has quit IRC | 20:05 | |
| sdague | update.py removes things from the list that aren't in g-r | 20:05 |
| sdague | it doesn't throw an error when it does it | 20:05 |
| Ajaeger | The requirements job stopped once it hit keystone-specs | 20:05 |
| *** pcm_ has joined #openstack-infra | 20:05 | |
| *** baoli has joined #openstack-infra | 20:05 | |
| *** rlandy has quit IRC | 20:05 | |
| fungi | sdague: tl;dr is that swiftclient added a requirement on futures. its havana backward-compat jobs didn't fail, even though futures is not in the havana branch of requirements (and update.py seems to have only modified the requirements it found in the global list, not removed any which weren't present in the global list) | 20:05 |
| *** pcm_ has quit IRC | 20:06 | |
| sdague | oh... ok | 20:06 |
| sdague | right, so we removed it | 20:06 |
| *** dimtruck is now known as zz_dimtruck | 20:06 | |
| fungi | er, maybe. actually now that you mention it, yes it didn't get installed | 20:06 |
| sdague | then it's backwards-compat job didn't try to use swiftclient to upload a file | 20:06 |
| Ajaeger | fungi: my reference above is to the post job - I see you talk about the gate... | 20:06 |
| sdague | because that's how we triggered it in grenade | 20:06 |
| *** e0ne has joined #openstack-infra | 20:07 | |
| fungi | so yes, i guess the job proved that swiftclient, while adding a new requirement, didn't actually exercise that requirement in the job | 20:07 |
| openstackgerrit | Spencer Krum proposed a change to openstack-infra/config: Conditionally manage puppetdb connection https://review.openstack.org/117377 | 20:07 |
| openstackgerrit | Spencer Krum proposed a change to openstack-infra/config: Fix depercation warnings in templates for unatttended upgrades https://review.openstack.org/116747 | 20:07 |
| openstackgerrit | Spencer Krum proposed a change to openstack-infra/config: Fixing deprecation warnings in logrotate module https://review.openstack.org/116752 | 20:07 |
| openstackgerrit | Spencer Krum proposed a change to openstack-infra/config: Enable puppet3 master bootstrap https://review.openstack.org/117604 | 20:07 |
| fungi | okay, i'll chalk that up to insufficient swift coverage in havana tempest jobs in that case | 20:07 |
| bdpayne | Are there any known issues with gerrit atm? | 20:07 |
| fungi | bdpayne: please describe perceived issues | 20:07 |
| sdague | fungi: that being said... exploding might not be a bad idea | 20:07 |
| *** zz_dimtruck is now known as dimtruck | 20:08 | |
| bdpayne | I just submitted a CR, and the automated gate jobs don't appear to be running | 20:08 |
| bdpayne | https://review.openstack.org/#/c/117596/ | 20:08 |
| fungi | sdague: so in that case, the only remaining open question is whether we should have backward-compat grenade jobs (does new proposed change fo fooclient master break upgrades from master-2 to master-1)? | 20:08 |
| fungi | bdpayne: i see 117596,1 in the check pipeline for 46 minutes at http://status.openstack.org/zuul/ | 20:09 |
| *** HeOS has quit IRC | 20:09 | |
| sdague | fungi: so... honestly, I think if we exploded on the requirement update that would be better | 20:09 |
| sdague | we only 'happened' to explode in grenade | 20:09 |
| fungi | bdpayne: still waiting on workers. thanks to rush week it'll be a while still | 20:09 |
| sdague | after havana is gone, we wouldn't have | 20:09 |
| sdague | because we now do the object upload with tempest client | 20:09 |
| sdague | so grenade found the issue by accident, not by intent | 20:10 |
| *** annegent_ has quit IRC | 20:10 | |
| sdague | and it was cryptic | 20:10 |
| sdague | if we explode on requirement update, then it's pretty clear what the issue is | 20:10 |
| Ajaeger | bdpayne: I have docs jobs for 6 hours in the queue ;( | 20:11 |
| bdpayne | oh wow... just that slow, thanks for checking fungi | 20:11 |
| bdpayne | heh, I feel silly for asking now :-) | 20:11 |
| fungi | bdpayne: yeah, there's been an almost perpetual backlog this week | 20:11 |
| *** ociuhandu has joined #openstack-infra | 20:11 | |
| fungi | sdague: fair enough. so i vaguely recall at one point we had a check for requirements in a change missing from global requirements but i want to say that deadlocked us on something... i don't recall what now though | 20:12 |
| sdague | yeh, jogo was hacking on a thing... | 20:12 |
| fungi | specifically a check in devstack jobs | 20:12 |
| *** dane_leblanc has quit IRC | 20:12 | |
| fungi | oh, or maybe it was hacking | 20:12 |
| fungi | i want to say something gummed us up from being able to land changes to openstack/requirements itself. anyway just trying to think through whether this would regress us into that state again | 20:13 |
| *** sballe has quit IRC | 20:14 | |
| *** lcheng_ has joined #openstack-infra | 20:14 | |
| fungi | and yeah, making update.py return nonzero on an extra requirement rather than just removing it does seem like a good way to plug the hole without adding additional jobs | 20:14 |
| sdague | yep | 20:15 |
| fungi | i'll go ahead and propose it and backports | 20:15 |
| sdague | let me see if I can hack that up real quick | 20:15 |
| fungi | oh, or feel free | 20:15 |
| *** mrmartin has quit IRC | 20:15 | |
| fungi | i can get back to making faces at what i'm now starting to think actually are hung building instances | 20:16 |
| fungi | nevermind. they seem to be actually returning in some way now | 20:16 |
| sdague | so... I think it's already there | 20:17 |
| sdague | but not backported | 20:17 |
| *** fifieldt has joined #openstack-infra | 20:17 | |
| openstackgerrit | Adam Gandelman proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1347692 https://review.openstack.org/117607 | 20:18 |
| uvirtbot` | Launchpad bug 1347692 in glance "gate failure with tempest.api.image.v1.test_images.CreateRegisterImagesTest.test_register_http_image" [Undecided,Confirmed] https://launchpad.net/bugs/1347692 | 20:18 |
| sdague | fungi: yeh, so it's in every branch .... except havana | 20:18 |
| fungi | ha. okay. maybe we just ignore it for a month and it'll go away at eol ;) | 20:18 |
| *** Ajaeger has quit IRC | 20:18 | |
| fungi | notmyname: so there was the eventual answer as to why it wasn't caught in the havana backward-compat jobs which ran on that change | 20:19 |
| sdague | well, I think I can get the backport done pretty quick | 20:20 |
| *** dane_leblanc has joined #openstack-infra | 20:24 | |
| sdague | fungi: https://review.openstack.org/117609 | 20:26 |
| *** denis_makogon has joined #openstack-infra | 20:26 | |
| *** eharney has quit IRC | 20:26 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Be able to generate math formulae with Sphinx mathpng base extension. https://review.openstack.org/117263 | 20:26 |
| *** ominakov has quit IRC | 20:30 | |
| *** e0ne has quit IRC | 20:30 | |
| *** lcheng_ has quit IRC | 20:31 | |
| *** otherwiseguy has quit IRC | 20:32 | |
| *** amotoki_ has quit IRC | 20:32 | |
| clarkb | fungi: would you think it silyl of me to request you suffix that variable with _package? | 20:33 |
| *** e0ne has joined #openstack-infra | 20:33 | |
| clarkb | also that is an oddly contentious review... | 20:34 |
| *** dizquierdo has joined #openstack-infra | 20:34 | |
| *** dkehn_ has joined #openstack-infra | 20:38 | |
| *** dustins has quit IRC | 20:38 | |
| *** ominakov has joined #openstack-infra | 20:39 | |
| *** dkehn_ has quit IRC | 20:40 | |
| *** dkehn_ has joined #openstack-infra | 20:40 | |
| *** eharney has joined #openstack-infra | 20:40 | |
| *** mmaglana has joined #openstack-infra | 20:40 | |
| *** dkehn has quit IRC | 20:41 | |
| *** dkehnx has quit IRC | 20:41 | |
| *** fifieldt_ has quit IRC | 20:42 | |
| *** e0ne has quit IRC | 20:42 | |
| *** dkehn has joined #openstack-infra | 20:42 | |
| fungi | yeah, i think it was just a matter of people getting confused by the params file there | 20:43 |
| openstackgerrit | Adrian Otto proposed a change to openstack-infra/config: OpenStack Containers Service https://review.openstack.org/115328 | 20:46 |
| *** e0ne has joined #openstack-infra | 20:46 | |
| ekarlso | pyay | 20:47 |
| ekarlso | one more project :P | 20:47 |
| *** e0ne has quit IRC | 20:49 | |
| *** HeOS has joined #openstack-infra | 20:50 | |
| *** jaypipes has quit IRC | 20:50 | |
| *** dizquierdo has quit IRC | 20:51 | |
| *** kgiusti has left #openstack-infra | 20:51 | |
| fungi | and clarkb sure, i'll update it to add a variable suffix | 20:52 |
| *** e0ne has joined #openstack-infra | 20:53 | |
| *** radez is now known as radez_g0n3 | 20:54 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Be able to generate math formulae with Sphinx mathpng base extension. https://review.openstack.org/117263 | 20:54 |
| jogo | sdague: what did I do? | 20:54 |
| openstackgerrit | Doug Hellmann proposed a change to openstack/requirements: Add missing oslo libs to projects list https://review.openstack.org/117620 | 20:56 |
| fungi | the bandersnatch resync is finally returning | 20:57 |
| jeblair | DEBUG:nodepool.NodePool: <AllocationGrant of 63 of devstack-precise-2-node from fake-provider3> | 20:57 |
| jeblair | there isn't even any demand for that one ^ :/ | 20:57 |
| fungi | huh | 20:57 |
| *** fifieldt_ has joined #openstack-infra | 20:59 | |
| jogo | fungi: isn't swiftclient in global dependencies | 21:00 |
| jogo | fungi: so it can quietly sneak anything in | 21:00 |
| clarkb | jeblair: it wanted to make sure devstack-precise-2-node was part of the party | 21:01 |
| clarkb | jogo: sdague propsed a backport to prevent quiet sneaking | 21:02 |
| ianw | jeblair: awesome! can you just pastebin a diff, in any state? | 21:02 |
| jogo | clarkb: ahh thats what sdague was talking about | 21:02 |
| jogo | clarkb: about the thing I did, but don't remember doing | 21:02 |
| *** pballand has quit IRC | 21:03 | |
| *** dkehn__ has joined #openstack-infra | 21:03 | |
| *** nelsnelson has quit IRC | 21:03 | |
| *** annegent_ has joined #openstack-infra | 21:04 | |
| *** dkehn__ has quit IRC | 21:05 | |
| *** dkehn__ has joined #openstack-infra | 21:05 | |
| *** dkehn_ has quit IRC | 21:05 | |
| *** dkehn has quit IRC | 21:05 | |
| *** andreaf has joined #openstack-infra | 21:06 | |
| *** pballand has joined #openstack-infra | 21:07 | |
| sdake | jeblair re infra, is there any code work that needs attention - eg some starter bugs to start learning? | 21:07 |
| *** dkehn has joined #openstack-infra | 21:07 | |
| clarkb | sdake: did you see my comment on 116802? | 21:07 |
| sdake | clarkb no let me check | 21:08 |
| clarkb | sdake: we should have bugs tagged low-hanging-fruit /me gets a link | 21:08 |
| clarkb | sdake: mostly trying to understand what was confusing there | 21:08 |
| sdake | clarkb didn't see that 116802, been only reviewing patches that have not had any review attention | 21:09 |
| *** esker has quit IRC | 21:09 | |
| *** annegent_ has quit IRC | 21:09 | |
| sdake | reading change now | 21:09 |
| sdake | oh nm, I did review this :) | 21:09 |
| clarkb | sdake: wait didn't you write that change or am I using a bad real name to irc nick mapping? | 21:09 |
| *** HeOS has quit IRC | 21:09 | |
| *** esker has joined #openstack-infra | 21:09 | |
| *** mfer has quit IRC | 21:10 | |
| *** esker has quit IRC | 21:10 | |
| clarkb | sdake: https://bugs.launchpad.net/openstack-ci/+bugs?field.tag=low-hanging-fruit the gerritbot and jjb bugs should be relatively easy to dive into | 21:10 |
| *** dangers_away is now known as dangers | 21:11 | |
| *** aysyd has quit IRC | 21:11 | |
| *** dkehn__ is now known as dkehnx | 21:12 | |
| *** armax has quit IRC | 21:13 | |
| *** annegent_ has joined #openstack-infra | 21:14 | |
| clarkb | sdake: zuul also has a separate bug tracker https://bugs.launchpad.net/zuul/ I don't see any low hanging fruit there but in general I think people like hacking on zuul if yo uare looking for less sysadminy stuff | 21:14 |
| sdake | clarkb I'd like to pitch in and learn how the whole system works long term (if thats a possible goal:) | 21:15 |
| sdake | but need to start on something I can chew off | 21:15 |
| clarkb | sdake: oh cool and it definitely is. | 21:15 |
| sdake | clarkb thanks for the links i'll start cracking on those | 21:15 |
| *** Sukhdev has joined #openstack-infra | 21:15 | |
| clarkb | sdake: a great place to start on the big picture is to spin up a service or two via puppet on some VMs. http://ci.openstack.org/sysadmin.html#making-a-change-in-puppet describes how you can do that | 21:16 |
| sdake | sweet i think I asked jeblair that exact question a couple days ago | 21:17 |
| sdake | I'll give that a go next week | 21:17 |
| *** dhellmann is now known as dhellmann_ | 21:17 | |
| ianw | sdague: i realise bashate is way way down on priority lists. little concerned patches don't seem to get much comment (month old stuff) which sucks for people trying their best to contribute. i missed when it got split out, who is driving it? | 21:18 |
| fungi | jogo: i actually think it can't sneak additional requirements in because 1. we have the requirements check at least against master, and 2. aside from havana we run a requirements sync on it in devstack before installing from source and then bail if it has additional requirements not in the global list | 21:18 |
| clarkb | ianw: gerrit will list the core members for you if you go to projects -> bashate -> access -> click on name of core group there -> members | 21:19 |
| clarkb | woo gerrit ui | 21:19 |
| clarkb | fungi: good point, it can only sneak changes to things it already snuck by | 21:19 |
| clarkb | maybe not even that? | 21:19 |
| fungi | and the havana hole is what sdague is addressing with the https://review.openstack.org/117609 backport | 21:20 |
| *** otherwiseguy has joined #openstack-infra | 21:20 | |
| *** r1chardj0n3s_afk is now known as r1chardj0n3s | 21:20 | |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/zuul: Add cache-hardlink argument to cloner https://review.openstack.org/117626 | 21:20 |
| *** pballand has quit IRC | 21:20 | |
| ianw | clarkb: yeah, i know that all devstack-core can +2, but nobody is :) | 21:21 |
| fungi | ianw: did you see the log sample jeblair posted earlier? | 21:22 |
| fungi | (for the nodepool post-mortem you're working on) | 21:22 |
| ianw | fungi: no, i don't think my scrollback went that far. you guys talk too much :) | 21:23 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/nodepool: WIP reproducible test https://review.openstack.org/117627 | 21:23 |
| fungi | speak of the debbil | 21:23 |
| jeblair | ianw: ^ that's what i'm working from right now | 21:23 |
| *** pballand has joined #openstack-infra | 21:23 | |
| jeblair | ianw: http://nodepool.openstack.org/allocator-failure.log | 21:23 |
| fungi | 1k lines of sql. pew pew lasers | 21:23 |
| jeblair | there's the production log | 21:23 |
| *** cnesa has quit IRC | 21:24 | |
| jeblair | fungi: it's auto-generated from the script, but i wanted to check it in anyway because it's random | 21:24 |
| jeblair | this is not getting merged :) | 21:24 |
| fungi | indeed, i gathered that | 21:24 |
| clarkb | 1k lines of sql o_O | 21:24 |
| pleia2 | so, cheese & wine party created a team who are fiddling with gerrit for Debian | 21:24 |
| fungi | pleia2: awesome! do they need any help/pointers? | 21:25 |
| fungi | pleia2: also, how goes the conf in general? good times? | 21:25 |
| ianw | jeblair: AllocationSubRequest for -136.551724138 (out of 40.0) of py3k-precise from hpcloud-b5> | 21:26 |
| jeblair | ianw, fungi: the only things i've found i can remove are: the tripleo provider is not needed and the number of building nodes in the db at startup can be lowered to 1000 (down from 2500) | 21:26 |
| *** terryw has joined #openstack-infra | 21:26 | |
| *** otherwiseguy has quit IRC | 21:26 | |
| *** e0ne has quit IRC | 21:27 | |
| jeblair | ianw, fungi: things that still seem to be necessary are: some number of building nodes on startup, image types that are not used, more than one provider | 21:27 |
| jeblair | ianw, fungi: i also tried scaling all the numbers by 0.1. that didn't work | 21:27 |
| ianw | jeblair: ok. what i was doing was similar, making fake entries in the db to get things out of sync. i think that negative there is what sends the allocator crazy | 21:27 |
| clarkb | jeblair: scaling numbers to fix the allocations? | 21:27 |
| *** e0ne has joined #openstack-infra | 21:27 | |
| jeblair | clarkb: no, i'm trying to make a smaller test case | 21:28 |
| sdake | clarkb re that change I submitted, I added a comment to explain my thoughts | 21:28 |
| sdake | probably the wording just needs improvement | 21:28 |
| clarkb | jeblair: oh I see | 21:28 |
| *** flwang_ has joined #openstack-infra | 21:28 | |
| clarkb | sdake: thanks looking | 21:28 |
| ianw | jeblair: https://review.openstack.org/#/c/109185/2/nodepool/tests/test_allocator.py <- that's input to the allocator that will send things crazy | 21:29 |
| jeblair | ianw, fungi: i also removed the ready nodes at startup, and while they are not strictly necessary, they seem to amplify the problem | 21:29 |
| clarkb | sdake: I didn't want to -1 because my familiarity with zuul probably biases me :) but I needed more info to understand | 21:29 |
| jeblair | ianw: yeah, the database is not "out of sync" | 21:29 |
| fungi | iiiiiinteresting | 21:29 |
| jeblair | ianw: there's nothing invalid about the data in the database | 21:29 |
| *** flwang_ has quit IRC | 21:29 | |
| clarkb | sdake: oh I see, they are using check and gate regardless of hte actual testing | 21:29 |
| sdake | clarkb just a fyi when I do reviews I either -1 or +1 so the folks know there is a decision on the bug | 21:29 |
| sdake | so hopefully that wont annoy you or others | 21:30 |
| ianw | jeblair: yeah, that was just my first thought. i hadn't considered the building / ready node interaction when starting up | 21:30 |
| fungi | sdake: i won't speak for anyone else, but i'm not easily annoyed by other people doing work ;) | 21:30 |
| pleia2 | fungi: just started for me, but good :) I just made contact with the folks doing the work so they'll let us know if they need pointers | 21:31 |
| jeblair | anyway, i'm heading back into my debugging hole | 21:31 |
| *** mfink has joined #openstack-infra | 21:32 | |
| sdake | clarkb I'm a firm believer if someone makes a mistake once others will repeat it :) | 21:32 |
| sdake | (this assumes people read the docs) | 21:32 |
| sdake | fungi sweet | 21:32 |
| clarkb | sdake: and they do, so now that I know this let me rereview | 21:32 |
| sdake | if you have some language to propose i'm happy to change | 21:33 |
| clarkb | noted | 21:33 |
| *** mmedvede has quit IRC | 21:33 | |
| *** SumitNaiksatam has quit IRC | 21:35 | |
| *** SumitNaiksatam has joined #openstack-infra | 21:36 | |
| *** andreaf has quit IRC | 21:37 | |
| *** dkranz has quit IRC | 21:37 | |
| *** Sincler has quit IRC | 21:37 | |
| *** andreaf has joined #openstack-infra | 21:37 | |
| fungi | sdake: i proposed one possibility in an inline comment | 21:38 |
| clarkb | sdake: fungi I did as well | 21:39 |
| *** mfink has quit IRC | 21:40 | |
| fungi | i'm cool with that suggestion too | 21:40 |
| hashar | clarkb: jeblair: I am unlikely to deploy the zuul cloner patches you wrote/reviewed to play test them. But they all seems fine to me :] So don't wait for me, I have already +1 ed all of them. | 21:41 |
| fungi | in general i agree that the existing text could be misinterpreted, just want to make sure we stick to consistent (for the project) terminology and not make any misstatements when attempting to clarify | 21:41 |
| *** tonytan4ever has quit IRC | 21:41 | |
| clarkb | hashar: I think the tests have pretty good coverage especially with your suggestions on making the tests more robust | 21:41 |
| *** ominakov has quit IRC | 21:42 | |
| *** andreaf has quit IRC | 21:43 | |
| *** mmaglana_ has joined #openstack-infra | 21:44 | |
| fungi | more often than not, when i look at pip download failures from our mirror, i see it's breaking while retrieving a setuptools wheel. i wonder what the connection is | 21:44 |
| clarkb | hashar: quick comment on your change | 21:44 |
| fungi | for example, this one currently afflicting a keystone change in the gate https://jenkins04.openstack.org/job/gate-keystone-python26/1400/console | 21:44 |
| clarkb | fungi: first package to download? | 21:44 |
| clarkb | setuptools is the thing you need before the other things | 21:44 |
| fungi | not even close to the first to download | 21:44 |
| clarkb | I have no idea then | 21:45 |
| fungi | python-ldap apparently wants a newer setuptools | 21:45 |
| hashar | clarkb: yeah that seems all good | 21:45 |
| *** mmaglana has quit IRC | 21:46 | |
| *** adalbas has quit IRC | 21:46 | |
| hashar | clarkb: jeblair: Iam going to be on paternity leave for a couple week pretty soon so will hardly be able to follow up :P | 21:46 |
| clarkb | hashar: congratulations | 21:46 |
| clarkb | hashar: we will miss your feedback but enjoy the time off | 21:46 |
| *** tonytan4ever has joined #openstack-infra | 21:47 | |
| *** tonytan4ever has quit IRC | 21:47 | |
| hashar | clarkb: all seems fine to me. I am not sure I can add anything more. The grenade test coverage, I am missing all the context and not familiar enough with devstack-gate to comment on it | 21:47 |
| fungi | yep, declares it in install_requires: http://python-ldap.cvs.sourceforge.net/viewvc/python-ldap/python-ldap/setup.py?revision=1.72&view=markup | 21:47 |
| hashar | but I am sure you guys will test it out for real before making the new job voting :] | 21:47 |
| *** alexpilotti has quit IRC | 21:47 | |
| * fungi sighs | 21:47 | |
| adam_g | does anyone know of a near-local URL we can use as an alternative for the horizon server here? https://review.openstack.org/#/c/32943/ this is causing a periodic horizon failure to fail image tests (https://launchpad.net/bugs/1347692) im thinking http://pypi.openstack.org | 21:48 |
| uvirtbot` | Launchpad bug 1347692 in glance "gate failure with tempest.api.image.v1.test_images.CreateRegisterImagesTest.test_register_http_image" [Undecided,Confirmed] | 21:48 |
| clarkb | fungi: oh and that will be serviced by easy_install | 21:48 |
| fungi | clarkb: yeah | 21:48 |
| fungi | everything you never wanted to know about python packaging but have been forced to find out anyway | 21:48 |
| clarkb | adam_g: something like http://tarballs.openstack.org/ironic-python-agent/coreos/ ? | 21:49 |
| clarkb | adam_g: that is a relatively large image though | 21:49 |
| *** annegent_ has quit IRC | 21:49 | |
| clarkb | you know what, this seems like the perfect thing to put in swift somewhere | 21:49 |
| adam_g | clarkb, we dont need any real data, the smaller the better i think | 21:50 |
| adam_g | its currently juts suing the horizon login page for the test image | 21:50 |
| clarkb | adam_g: you could just grab an sdist off of pypi.openstack.org | 21:50 |
| clarkb | or does it need to be a actual image of some sort | 21:50 |
| adam_g | clarkb, just the index of pypi.openstack.org would work | 21:50 |
| clarkb | adam_g: the index is probably pretty big as it lists all the things but ya we could do somethign like that | 21:51 |
| clarkb | adam_g: use https://git.openstack.org/cgit | 21:51 |
| adam_g | clarkb, http://pypi.openstack.org is minimal | 21:51 |
| clarkb | since we are already slamming those servers and have set them up to scale horizontally | 21:51 |
| clarkb | adam_g: oh you mean the root root | 21:51 |
| clarkb | ya, I like the idea of using the git farm | 21:51 |
| clarkb | fungi: ^ | 21:51 |
| *** dimtruck is now known as zz_dimtruck | 21:52 | |
| *** zz_dimtruck is now known as dimtruck | 21:53 | |
| *** _nadya_ has joined #openstack-infra | 21:54 | |
| *** e0ne has quit IRC | 21:56 | |
| *** chuckC has quit IRC | 21:56 | |
| dstufft | fungi: clarkb do y'all auto update bandersnatch in any way | 21:57 |
| *** mriedem has quit IRC | 21:57 | |
| dstufft | like, bandersnatch itself | 21:57 |
| dstufft | not the mirror | 21:57 |
| fungi | clarkb: me too | 21:57 |
| fungi | dstufft: i'll check | 21:57 |
| *** Sincler has joined #openstack-infra | 21:58 | |
| *** laca has quit IRC | 21:58 | |
| fungi | dstufft: apparently not at the moment... http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/manifests/static.pp#n242 | 21:58 |
| *** mjturek has left #openstack-infra | 21:58 | |
| fungi | dstufft: looks like we're on 1.4 and 1.5 is the latest | 21:58 |
| *** _nadya_ has quit IRC | 21:59 | |
| clarkb | dstufft: should we upgrade? | 21:59 |
| dstufft | fungi: I'm working on fixing the name mismatch on mirror thing... and I have a problem about how to handle the incompatability... the change requires updates in both pip and bandersnatch, however if you use an older pip (< 1.6) or an older bandersnatch (< whatever version contains the fix) you'll get the error that you're seeing now basically everywhere | 21:59 |
| fungi | the two items in the release notes for 1.5 make it seem not urgent | 21:59 |
| openstackgerrit | Oscar Romero (homeless) proposed a change to openstack-infra/jeepyb: Adding 'has-local-git' parameter to projects config https://review.openstack.org/117636 | 21:59 |
| dstufft | e.g. if you update your bandersnatch mirror to a version that contains the fix, but you use pip 1.5 with it, then pip will do the "you typed the name in wrong" thing | 22:00 |
| fungi | dstufft: i think as long as we know to expect it, we can cope. also we could consider switching that from "present" to "latest" i think and just drink from the release firehose | 22:00 |
| *** e0ne has joined #openstack-infra | 22:00 | |
| *** hashar has quit IRC | 22:01 | |
| fungi | oh yikes... we're using pip 1.0 with bandersnatch :/ | 22:01 |
| dstufft | Q.Q | 22:01 |
| fungi | pip 1.0 from /usr/lib/python2.7/dist-packages (python 2.7) | 22:01 |
| clarkb | wow | 22:01 |
| fungi | ii python-pip 1.0-1build1 alternative Python package installer | 22:01 |
| dstufft | I need to think about this some more about how to do this without breaking everyone using mirrors currently :[ | 22:01 |
| fungi | courtesy of ubuntu precise | 22:01 |
| fungi | so... we may want to noodle on that a bit | 22:02 |
| openstackgerrit | Steven Dake proposed a change to openstack-infra/zuul: Clarify docs for optional template pipelines https://review.openstack.org/116802 | 22:02 |
| *** andreykurilin has joined #openstack-infra | 22:02 | |
| clarkb | fungi: are you in the neutron channel? | 22:02 |
| clarkb | salv-orlando needs you to backup why a /0 subnet is bad ;) | 22:03 |
| fungi | clarkb: now i am | 22:03 |
| *** camunoz has joined #openstack-infra | 22:03 | |
| *** laca has joined #openstack-infra | 22:03 | |
| *** dmsimard is now known as dmsimard_away | 22:03 | |
| openstackgerrit | Adam Gandelman proposed a change to openstack-infra/devstack-gate: Point TEMPEST_HTTP_IMAGE at git.openstack.org https://review.openstack.org/117637 | 22:03 |
| fungi | clarkb: also, the bandersnatch resync dstufft suggested seems to have properly populated that flask-wtc wheel now. no idea why bandersnatch failed to properly do that in a normal update though | 22:04 |
| clarkb | fungi: that is great news | 22:04 |
| clarkb | fungi: is it done syncing? | 22:04 |
| clarkb | adam_g: out of curiousity is horizon failing and we expect it to fail more than some external server? | 22:05 |
| clarkb | I guess it is worth a go and the git mirrors seem pretty darn solid | 22:05 |
| *** denis_makogon has quit IRC | 22:05 | |
| salv-orlando | clarkb: I thought this was a no-brainer. Did not really expect that! | 22:05 |
| fungi | clarkb: seems to be done, yeah | 22:06 |
| *** jgrimm is now known as zz_jgrimm | 22:06 | |
| adam_g | clarkb, yes. its failing because of another obscure gate issue that is on peoples radar but not yet fixed (https://bugs.launchpad.net/horizon/+bug/1345955). it would be great to isolate that from images | 22:06 |
| uvirtbot` | Launchpad bug 1345955 in horizon "KeyError: <SassRule <RuleAncestry ()>, 0 props> Failure" [Medium,Confirmed] | 22:07 |
| *** linkid has left #openstack-infra | 22:07 | |
| openstackgerrit | K Jonathan Harker proposed a change to openstack-infra/config: Add single_use_slave coverage to the apply tests https://review.openstack.org/117389 | 22:07 |
| clarkb | adam_g: gotcha, +2 noted we could also do another http server on say port 8080 and use that | 22:07 |
| clarkb | but I am happy to try git.o.o since the jobs depend on it already | 22:08 |
| adam_g | yeah | 22:08 |
| *** amcrn has quit IRC | 22:08 | |
| adam_g | setting up a whole new server seems like more work than its worth | 22:08 |
| mattoliverau | Morning | 22:09 |
| clarkb | morganfainberg_Z: good morning | 22:09 |
| *** e0ne has quit IRC | 22:09 | |
| clarkb | er | 22:09 |
| clarkb | mattoliverau: o/ | 22:09 |
| *** lcheng_ has joined #openstack-infra | 22:09 | |
| clarkb | is feature freeze next week? | 22:09 |
| *** eharney has quit IRC | 22:10 | |
| *** ominakov has joined #openstack-infra | 22:11 | |
| adam_g | thursday | 22:11 |
| clarkb | thanks | 22:11 |
| *** pradk has quit IRC | 22:12 | |
| jogo | how do you make a note in the meeting notes? | 22:13 |
| jogo | we tried '#note' | 22:13 |
| jogo | ohh #info | 22:13 |
| *** tsufiev has quit IRC | 22:14 | |
| *** lcheng_ has quit IRC | 22:14 | |
| *** dteselkin has quit IRC | 22:14 | |
| clarkb | I was trying to get a listing btu apparently help is not a thing | 22:14 |
| *** amcrn has joined #openstack-infra | 22:14 | |
| clarkb | jogo: when you start a meeting the useful actions are listed | 22:14 |
| clarkb | which isn't a completel list | 22:14 |
| *** dteselkin has joined #openstack-infra | 22:15 | |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/config: Add the tempest-lib project to openstack https://review.openstack.org/117640 | 22:15 |
| jogo | clarkb: oh its not complete list lol | 22:15 |
| jogo | anyway I think '#info' is what I am looking for | 22:15 |
| clarkb | yup | 22:16 |
| *** ominakov has quit IRC | 22:16 | |
| fungi | wow... we merged 100 changes to openstack/openstack submodules on tuesday | 22:17 |
| fungi | 85 so far today, with a couple hours to go | 22:17 |
| *** tsufiev has joined #openstack-infra | 22:17 | |
| dstufft | clarkb: fungi well, if anything, the "package rename causes things to blow up" should be fixed now | 22:17 |
| *** fifieldt has quit IRC | 22:17 | |
| *** amcrn has quit IRC | 22:17 | |
| *** HeOS has joined #openstack-infra | 22:17 | |
| *** paulrad has quit IRC | 22:18 | |
| fungi | dstufft: oh? i take it that was fixed with aggressive cache expiration for reditects? | 22:18 |
| fungi | redirects | 22:18 |
| clarkb | hrm no dims? | 22:18 |
| clarkb | dstufft: cool and thank you | 22:18 |
| dstufft | fungi: Nope, simple always normalizes now, so "Django" is /simple/django/ | 22:18 |
| dstufft | so if you delete "Django" and then upload "django", the simple URL doesn't change | 22:18 |
| fungi | dstufft: oh! yes, per your pkg-sig post | 22:18 |
| sdake_ | fungi works for me | 22:19 |
| fungi | dstufft: will we need to rebuild our mirror for that then? | 22:19 |
| dstufft | fungi: no, that's not reflected in bandersnatch at all | 22:19 |
| sdake_ | clarkb i'll have a look at your suggestion, but I need to run out for an errand | 22:19 |
| dstufft | bandersnatch computes it's own URL using the package name | 22:20 |
| dstufft | so it'll see it needs to sync "Django", it | 22:20 |
| dstufft | it'll fetch /simple/Django/, get redirected and get the content of /simple/django, and then save it at /simple/Django/index.html | 22:20 |
| fungi | dstufft: oh, okay. this is just rewriting requests internal to pypi without using a redirect? | 22:20 |
| fungi | oh, so still redirecting | 22:20 |
| fungi | or transparently? | 22:20 |
| dstufft | yea still redirecting, but everything will still work via redirect | 22:21 |
| clarkb | sdake_: I am +2 on that zuul change now | 22:21 |
| dstufft | that doesn't fix your "pip install flask-wtf from a bandersnatch mirror doesn't work" problem | 22:21 |
| clarkb | looks like you used fungi's suggsetion | 22:21 |
| dstufft | that needs changes to both pip and bandersnatch :/ | 22:21 |
| clarkb | dstufft: I can't help but think that package name is really appropriate here | 22:21 |
| *** dimtruck is now known as zz_dimtruck | 22:22 | |
| sdake_ | clarkb cool no need to overdo it then :) | 22:22 |
| sdake_ | clarkb I didn't see your suggestion until after I submitted the change | 22:22 |
| dstufft | I can't think of any way to fix that which doesn't end up being "release pip 1.6 and the fixed bandersnatch at the same time, and tell people they can't mix use pip 1.5 with bandersnatch new | 22:22 |
| sdake_ | either one works | 22:23 |
| dstufft | ooo | 22:23 |
| sdake_ | we may be too hopeful that people actually read docs - I know I don't :) | 22:23 |
| dstufft | actually | 22:23 |
| dstufft | I can duplicate entries! | 22:23 |
| dstufft | I can make an entry for *both* /srv/pypi/simple/Django/index.html *and* /srv/pypi/simple/django/index.html | 22:23 |
| fungi | dstufft: though actually, our "pip install flask-wtf from a bandersnatch mirror doesn't work" problem turned out to be that bandersnatch never downloaded the wheel for the latest version (a maintenance resync of the mirror like you suggested fixed it) | 22:23 |
| fungi | dstufft: so we were serving it as a link in the index, but the link went to a 404 | 22:24 |
| dstufft | or maybe it wasn't flask-wtf, whichever one was "unless we use the exact name that the person used on PyPI we get an error about external urls" | 22:24 |
| fungi | dstufft: yeah, i get which problem you mean, just don't remember where we saw it happen | 22:25 |
| fungi | but cool | 22:25 |
| clarkb | fungi: dstufft we didn't hit the flask-wtf case issue because it failed the other way, but now that it is synced we will fall in the way dstufft describes | 22:25 |
| *** chuckC has joined #openstack-infra | 22:26 | |
| *** stevemar has quit IRC | 22:27 | |
| *** dangers is now known as dangers_away | 22:27 | |
| clarkb | fungi: any idea why /etc/puppet/modules/storyboard doesn't have jeblairs change to that module in it yet? | 22:31 |
| *** amotoki has joined #openstack-infra | 22:33 | |
| clarkb | I think install_modules.sh doesn't actually update the tracking branch | 22:33 |
| *** dkranz has joined #openstack-infra | 22:33 | |
| clarkb | and we want to do origin/master instead of master | 22:34 |
| clarkb | fungi: ^ can you take a quick look at that and see if it makes sense to you? Note that it does a git fetch then a git checkout | 22:34 |
| *** markmcclain has quit IRC | 22:34 | |
| *** sandywalsh has quit IRC | 22:34 | |
| *** morganfainberg_Z is now known as morganfainberg | 22:34 | |
| clarkb | fungi: might also be related to the rev listing | 22:34 |
| fungi | clarkb: you mean for jeblair's change which wipes out /etc/puppet/modules? | 22:36 |
| clarkb | I think i have convinced myself patch incoming | 22:36 |
| clarkb | fungi: wasn't that for testing only? | 22:36 |
| clarkb | fungi: can you link me to that change | 22:37 |
| fungi | clarkb: no, i thought it was really for reals | 22:37 |
| fungi | clarkb: https://review.openstack.org/#/c/117365/1 | 22:37 |
| clarkb | ya that is just for testing | 22:38 |
| clarkb | on the slaves | 22:38 |
| *** amotoki has quit IRC | 22:38 | |
| fungi | clarkb: oh, i misunderstood what you meant by just for testing | 22:39 |
| *** dkehn_ has joined #openstack-infra | 22:39 | |
| *** leifmadsen has quit IRC | 22:39 | |
| *** HeOS has quit IRC | 22:40 | |
| fungi | clarkb: so asking why https://review.openstack.org/116971 isn't actually on the storyboard.o.o server then? | 22:40 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Update git puppet modules properly. https://review.openstack.org/117643 | 22:40 |
| clarkb | fungi: ^ | 22:40 |
| clarkb | no ci-puppetmaster.o.o/etc/puppet/modules/storyboard id old | 22:41 |
| fungi | huh... /etc/puppet/modules on storyboard.o.o is empty | 22:41 |
| *** gondoi is now known as zz_gondoi | 22:41 | |
| ianw | jeblair: i can get those crazy allocations coming out with http://paste.openstack.org/show/102034/ , i was trying to replicate the exact numbers to get a feel for what was wrong but it's too much of a simultaneous equation | 22:41 |
| clarkb | fungi: it should be on that server | 22:41 |
| clarkb | fungi: the issue was we told it to track master which we are but master is never updated | 22:41 |
| fungi | aha | 22:41 |
| *** dkehn_ has quit IRC | 22:42 | |
| clarkb | fungi: so I am telling it to follow origin/master which remote update will keep up to date | 22:42 |
| *** dkehn has quit IRC | 22:42 | |
| fungi | clarkb: yep, origin/master not (local)master. got it | 22:42 |
| fungi | we're spoiled by puppet-vcsrepo | 22:42 |
| *** dkehn_ has joined #openstack-infra | 22:42 | |
| *** dkehnx has quit IRC | 22:42 | |
| clarkb | yup | 22:42 |
| ianw | jeblair: results http://paste.openstack.org/show/102036/ | 22:43 |
| clarkb | anyways I think that will do it, and my change should go in before the storyboard rabbitmq disablement | 22:43 |
| clarkb | fungi: also it may make jeblair's change a noop | 22:43 |
| clarkb | fungi: actually maybe not because we need to do integration testing which means clone a zuul thing instead of origin/master | 22:43 |
| *** dkehn has joined #openstack-infra | 22:44 | |
| clarkb | but this should fix the ci-puppetmaster module state | 22:44 |
| fungi | yeah, i think the rm -rf is still necessary, since we won't reflect things like module removals otherwise | 22:45 |
| jeblair | ianw: why does it say "AllocationSubRequest" instead of "AllocationGrant" when you print the grants? | 22:47 |
| jeblair | ianw: or did you just paste printing the subrequests? | 22:47 |
| clarkb | nibalizer: why do your fix deprecation warnings changes all depend on each other? | 22:47 |
| clarkb | nibalizer: do they actually need to depend on each other? | 22:47 |
| *** doug-fish has left #openstack-infra | 22:48 | |
| clarkb | nibalizer: I aks because I want to -1 a change in the middle ... and that will disrupt the chain | 22:48 |
| clarkb | nibalizer: maybe I will just have you document the thing at the end of the stack | 22:48 |
| clarkb | nibalizer: I went ahead and did it that way | 22:50 |
| clarkb | hrm it is far too late for Ajaeger. Any opinions on whether or not openstack-doc-tools should follow global requirements? my feeling is no for the same reasons we don't | 22:51 |
| *** ominakov has joined #openstack-infra | 22:52 | |
| *** andreykurilin has quit IRC | 22:55 | |
| nibalizer | clarkb: i dont think they need to | 22:59 |
| nibalizer | but i need the commit at the end to depend on all of them | 22:59 |
| clarkb | nibalizer: gotcha | 22:59 |
| nibalizer | the make puppet 3 master go (no for realzies this time) | 22:59 |
| nibalizer | is it possible to have a multi dependency without building a chain like that? | 22:59 |
| clarkb | nibalizer: no | 22:59 |
| nibalizer | baw | 22:59 |
| *** joools has joined #openstack-infra | 23:01 | |
| openstackgerrit | A change was merged to openstack-infra/config: Remove duplicate barbican job https://review.openstack.org/116959 | 23:02 |
| dstufft | clarkb: fungi https://bitbucket.org/pypa/bandersnatch/pull-request/7/prefer-normalized-directory-names-but-keep/diff part 1 of 2 for fixing the mirrors when you don't have the exact name | 23:02 |
| *** terryw has quit IRC | 23:02 | |
| dstufft | part 2 will be inside pip | 23:02 |
| *** mriedem has joined #openstack-infra | 23:02 | |
| *** otherwiseguy has joined #openstack-infra | 23:04 | |
| jeblair | the problem happens when the number of available nodes for a provider is negative. | 23:04 |
| jeblair | when the history allocator detects a neglected request, it immediately allocates min(request, available) to that request, even if the request is 0 | 23:04 |
| jeblair | this means that it will attempt to grant a negative value. there's a check that prevents that from becoming an actual grant, however, it does affect the available count. it would increase it if the value is negative | 23:04 |
| jeblair | if the neglected image it has chosen to grant nodes to (even though it requested none) is a subnode image, the available node count can increase at 2x the normal rate | 23:04 |
| jeblair | when it gets around to servicing an image that actually did request something, it believes there are many nodes available. | 23:04 |
| jeblair | this is far less likely to happen with the standard allocator because 0-node requests end up clamped to 0 regardless of availablity due to the ratio calculation | 23:04 |
| jeblair | clarkb, fungi, ianw: ^ | 23:04 |
| clarkb | fun | 23:06 |
| clarkb | jeblair: is that something we can fix in the history allocator with better bounds checking? | 23:06 |
| fungi | got it | 23:06 |
| fungi | so ianw was on the right track with his observation that negative availability was causing the allocator to go wild | 23:07 |
| jeblair | yep, but now we know why, and can verify the fix | 23:07 |
| *** dane_leblanc has quit IRC | 23:07 | |
| fungi | exactly | 23:07 |
| nibalizer | clarkb: so you said you're not preturbed by puppet not running on the puppet master | 23:07 |
| nibalizer | to your knowledge, does it run successfully? | 23:08 |
| jeblair | the only potential bug i found in the history allocator per se, is i think there may be a code path that would cause it to process a request twice, but only if the availability is 0 | 23:08 |
| *** Sincler has quit IRC | 23:08 | |
| clarkb | nibalizer: it is runnin on the pupept master | 23:08 |
| clarkb | nibalizer: which is why I was confused | 23:08 |
| jeblair | however, now that i really understand this, i think it's worth noting that the history allocator will pretty much run all the time in our environment | 23:08 |
| nibalizer | oh right | 23:08 |
| *** atiwari has quit IRC | 23:09 | |
| fungi | nibalizer: http://puppetdb.openstack.org/node/ci-puppetmaster.openstack.org | 23:09 |
| nibalizer | cus the puppetmaster.o.o isnt real and the real puppet master is ci-puppetmaster | 23:09 |
| nibalizer | sorry, /me derped | 23:09 |
| jeblair | so we'll go from completely proportional to something more like "every time a node is available, launch as many nodes of the least recently launch image type as possible" | 23:09 |
| fungi | hrm, true, since we're always at a deficit for all node types under load | 23:10 |
| clarkb | jeblair: which will keep fedora nodes aroudn but may affect precise/trusty/noesthatrunlotsoftests | 23:10 |
| jeblair | yes. the scales will tip in fedora's favor | 23:10 |
| fungi | it'll turn into more of a round-robin allocator, best case? | 23:10 |
| jeblair | fungi: with random aspects if more than one node is available in a cycle | 23:11 |
| clarkb | which may be what we want as far as reporting responsiveness goes | 23:11 |
| fungi | ahh, yeah | 23:11 |
| clarkb | its not fairest | 23:11 |
| *** terryw has joined #openstack-infra | 23:12 | |
| jeblair | and it will make bad decisions like "spin up a fedora node on the only provider that can provide a wacky-kernel node" | 23:12 |
| *** otherwiseguy has quit IRC | 23:12 | |
| clarkb | right | 23:12 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Make list of open reviews bold https://review.openstack.org/117588 | 23:12 |
| dstufft | clarkb: fungi and here's part 2! https://github.com/pypa/pip/pull/2001 | 23:13 |
| jeblair | if we're okay with that, we may as well simplify it and make the whole thing round robin or something else | 23:13 |
| clarkb | jeblair: so I think the actual benefit to this change is writing the next chagne that hasn't been written yet to pid loop the whole thing | 23:14 |
| jeblair | (i'm not sure i am; i think i'd prefer to see the existing allocator tweaked, but i'm biased) | 23:14 |
| ianw | jeblair: awesome digging | 23:14 |
| clarkb | jeblair: maybe we should write that chaneg too then switch to it | 23:14 |
| clarkb | jeblair: as that should deal with the issues you point out much better | 23:14 |
| jeblair | clarkb: you volunteering? :) | 23:15 |
| *** Sukhdev has quit IRC | 23:15 | |
| clarkb | maybe? it is probably a non trivial change and with feature freeze won't get to it asap but it would be a fun thing to hack on | 23:15 |
| jeblair | clarkb: i'm not really sure how a pid loop applies here | 23:16 |
| clarkb | jeblair: it would measure rates of changes for each of the inputs over time and adjust its outputs to converge on optimal node allocation | 23:17 |
| *** linkid has joined #openstack-infra | 23:17 | |
| jeblair | clarkb: what's the optimal node allocation? | 23:17 |
| *** david-lyle has quit IRC | 23:17 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for bug 1362347 https://review.openstack.org/117358 | 23:17 |
| uvirtbot` | Launchpad bug 1362347 in neutron "neutron-ns-meta invokes oom-killer during gate runs" [High,New] https://launchpad.net/bugs/1362347 | 23:17 |
| clarkb | jeblair: thats the tricky thing. when flying a quadropter its make go over to position X. with nodepool it may be a lot more neubulous than that. But probably something based on change throughput | 23:17 |
| fungi | i'm starting to think this isn't going to be solved without nodepool considering more information from zuul | 23:18 |
| clarkb | which would have to be weighted because some jobs run much longer than others | 23:18 |
| clarkb | jeblair: the desired output is that changes don't sit in a queue for 35 hours waiting on a single node | 23:18 |
| ianw | jeblair: i'd thought as a first step it would work by clamping the allocation given to nodes that missed out last time. as you say, we currently give it all available. any other value seemed arbitrary (give half of what's available, 75% of what's available, etc) | 23:18 |
| clarkb | jeblair: we can stop allocating trusty nodes periodically to make that happen using a pid loop in a much smarter way than round robin | 23:18 |
| jeblair | ianw: this loop cycles every few seconds, often with no change in available nodes, so we'd pretty much be hitting that case continuously for all node types | 23:19 |
| jeblair | ianw: at least, when busy. and of course, when busy is the only time any of this matters :) | 23:20 |
| clarkb | but I think that change is very non trivial and will require a lot more thought than I have just given it | 23:21 |
| *** annegent_ has joined #openstack-infra | 23:22 | |
| *** dmsimard_away is now known as dmsimard | 23:23 | |
| clarkb | nibalizer: can you abandon 113627 and open a bug to remove that modules from the forge if that is what we are doing | 23:24 |
| clarkb | nibalizer: is removing from the forge something that is acceptable? if someone did that on pypi I would be unhappy | 23:24 |
| *** annegent_ has quit IRC | 23:25 | |
| *** melwitt has quit IRC | 23:25 | |
| nibalizer | sure | 23:26 |
| clarkb | thanks | 23:26 |
| nibalizer | uh idunno it could go either way | 23:26 |
| nibalizer | id just really rather people not expect that code to be code or to work at all | 23:26 |
| nibalizer | and i want the whole puppet dashboard to diaf | 23:26 |
| clarkb | nibalizer: right, but anyone using it today will break | 23:26 |
| nibalizer | oh hrm | 23:27 |
| *** terryw has quit IRC | 23:27 | |
| nibalizer | so no one should be using openstack-infra/puppet-dashboard | 23:27 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Use direct gerrit contactstore to avoid CDN issues https://review.openstack.org/117650 | 23:27 |
| nibalizer | but... anyone who has consumed all of infra and built their own ci | 23:27 |
| clarkb | "should" | 23:27 |
| nibalizer | may have pulled it in as a dependency | 23:27 |
| clarkb | right | 23:27 |
| fungi | that ^ is tested and is something i'd like in place before the gerrit restart saturday, btw | 23:27 |
| clarkb | fungi: reviewing now | 23:28 |
| clarkb | fungi: they must have decided that the CDN wasn't going to whitelist that traffic? | 23:28 |
| nibalizer | clarkb: yea in fact we at HP try to pull that in | 23:28 |
| nibalizer | :/ | 23:29 |
| nibalizer | okay i guess we'll just leave it there forever | 23:29 |
| nibalizer | maybe i should add a commit to to the readme to say "this one is dead, use something else" | 23:29 |
| clarkb | This is not the module you are looking for | 23:30 |
| fungi | clarkb: i think they decided that it would be easier to do this than try to bend the cdn to their will on it | 23:30 |
| wenlock | nibalizer forj.io does | 23:31 |
| clarkb | we don't care about stackforge projects are potentially trademark infringeing right? | 23:31 |
| wenlock | nibalizer but we use a freeze of it, so it should be fine to nuke, we'll switch as we upgrade to newer stuff | 23:31 |
| clarkb | fungi: wfm | 23:31 |
| wenlock | if it can be avoided to remove from puppetlabs forge, that would be nice | 23:31 |
| wenlock | clarkb im trying to consume the latest install_module.sh , but i keep getting this error on the new modules http://paste.forj.io/show/23/ | 23:33 |
| nibalizer | yae i guess we'll keep it there | 23:33 |
| nibalizer | ya thats a known issue | 23:34 |
| nibalizer | crinkle: has a patch in to fix it | 23:34 |
| clarkb | nibalizer: but no one will tes tit... | 23:34 |
| clarkb | that is an unfortunate tyop | 23:34 |
| wenlock | clarkb i think your using stdlib 4.3.2 vs 3.2.0 | 23:34 |
| clarkb | wenlock: correct | 23:34 |
| wenlock | clarkb can i make a push to set stdlib to 4.3.2? | 23:34 |
| clarkb | wenlock: as nibalizer says there is a patch to change it but I haven't heard if anyone has tested it | 23:34 |
| nibalizer | clarkb: what do you consider sufficient testing? i.e. the puppet apply tests aren't good enough for you | 23:34 |
| clarkb | wenlock: that is already done | 23:34 |
| clarkb | nibalizer: no, bceause we don't actually reinstall the modules properly in those tests today | 23:35 |
| nibalizer | ohh | 23:35 |
| nibalizer | lost to fix :) | 23:35 |
| wenlock | clarkb cool.... what does "testing it" mean? | 23:35 |
| clarkb | nibalizer: jeblair has a patch to change that | 23:35 |
| nibalizer | if i got you annecdotal evidence that a few different nodes all work fine with that patch would that give you enough confidence? | 23:35 |
| clarkb | wenlock: nibalizer: isntall the module then run some puppet applies. I am not super worried about it as stdlib seems to have a relatively stable api | 23:35 |
| clarkb | but it touches all the things so I am wary | 23:35 |
| openstackgerrit | A change was merged to openstack-infra/config: Announce tooz events in #openstack-state-management https://review.openstack.org/116844 | 23:36 |
| crinkle | that patch is now outdated since the latest version of stdlib is 4.3.2 not 4.2.1 >.> | 23:36 |
| clarkb | nibalizer: ya that would be good, things like gerrit, and slave manifests | 23:36 |
| wenlock | agreed.... but making it 4.3.2... i think you must already be using it no? | 23:36 |
| clarkb | wenlock: we are not | 23:36 |
| clarkb | we are using 3.2.o | 23:36 |
| crinkle | I have played with it and convinced myself that it works, and stdlib 4 is completely backwards compatible with stdlib 3 and still tested on puppet 2.7 | 23:36 |
| clarkb | crinkle: oh good I didn't know that | 23:37 |
| wenlock | clarkb but the script as is in master leaves behind 4.3.2 | 23:37 |
| clarkb | wenlock: it doesn't if you installed 3.2.0 when 3.2.0 was the thing | 23:37 |
| wenlock | ahhh gotcha ;D | 23:37 |
| nibalizer | wenlock: have you looked at puppetboard or puppet explorer? | 23:37 |
| clarkb | crinkle: nibalizer: maybe the thing to do is pick a time when you all can watch the change go in | 23:37 |
| wenlock | so fresh installs like our own are seeing 4.3.2 | 23:37 |
| nibalizer | to replace the dashboard | 23:37 |
| clarkb | I can approve and we deal with fallout if any | 23:37 |
| crinkle | clarkb: works for me | 23:38 |
| nibalizer | wfm | 23:38 |
| clarkb | I didn't realize that it was supposed to be backward compat. why bump the version number? | 23:38 |
| clarkb | wenlock: yes fresh installs... | 23:38 |
| clarkb | crinkle: nibalizer: does tomorrow work or are you fargoing? | 23:38 |
| crinkle | there was a lot of fuss about it, i can find the gh issue if you want | 23:38 |
| clarkb | crinkle: nah its ok | 23:38 |
| clarkb | crinkle: its incredibly misleading though :) | 23:38 |
| crinkle | yes | 23:38 |
| nibalizer | clarkb: tomorrow afternoon/evening could work | 23:38 |
| nibalizer | morning is a bit busted for me | 23:39 |
| *** signed8bit is now known as signed8bit_ZZZzz | 23:39 | |
| clarkb | nibalizer: I was thinking morning bceause EFRIDAY | 23:39 |
| wenlock | clarkb, ok that works for us, i'll ignore the issue and upgrade on our end. i want to propose a modules.env update as well | 23:39 |
| nibalizer | im in a car in the morning | 23:39 |
| crinkle | clarkb: i can submit a revised patch to bump it to 4.3.2 instead of 4.2.1 | 23:39 |
| crinkle | since that might make more sense | 23:39 |
| clarkb | crinkle: cool lets do that | 23:39 |
| crinkle | mmk | 23:39 |
| nibalizer | crinkle: ++ | 23:39 |
| wenlock | clarkb so we can alternatively source modules from a file | 23:39 |
| clarkb | and play by ear tomorrow | 23:39 |
| nibalizer | ok | 23:39 |
| wenlock | clarkb i think this way people can consume install_modules.sh as is better in other projects | 23:40 |
| *** amcrn has joined #openstack-infra | 23:40 | |
| clarkb | wenlock: there is a jeblair change you should be aware of. let me find it | 23:41 |
| openstackgerrit | Colleen Murphy proposed a change to openstack-infra/config: Update version of puppetlabs-stdlib in install_modules.sh https://review.openstack.org/109351 | 23:41 |
| *** ominakov has quit IRC | 23:41 | |
| clarkb | wenlock: https://review.openstack.org/#/c/117323/1/install_modules.sh that one | 23:41 |
| *** ominakov has joined #openstack-infra | 23:43 | |
| wenlock | clarkb not much excitement there... just optionally installing storyboard | 23:43 |
| wenlock | clarkb am i missing something? | 23:43 |
| clarkb | wenlock: the optional installation is the thin gthing | 23:43 |
| clarkb | *is the thing | 23:43 |
| openstackgerrit | A change was merged to openstack-infra/config: Create merlin repo https://review.openstack.org/110325 | 23:43 |
| clarkb | the special include stuff sould accomodate that | 23:43 |
| wenlock | ah yes! | 23:44 |
| wenlock | it will | 23:44 |
| wenlock | clarkb check it out https://review.forj.io/#/c/1659/ this is what im thinking | 23:45 |
| nibalizer | i dont really get this either clarkb | 23:45 |
| jeblair | clarkb, fungi, ianw: i have a vague idea about having the history tracker record number of nodes allocated since the last allocation of a given type, rather than cycles since, to decouple it from time | 23:45 |
| jeblair | clarkb, fungi, ianw: and then possibly using that in a ratio to gradually shift priority of neglected nodes | 23:45 |
| jeblair | clarkb, fungi, ianw: but first i'm going to evaluate ianw's bug fixes with the new knowledge and tests | 23:45 |
| clarkb | jeblair: that would be a form of pid I think. so ++ | 23:46 |
| clarkb | nibalizer: get what? | 23:46 |
| nibalizer | the conditional logic jeblair is adding | 23:46 |
| clarkb | nibalizer: oh its there so that we can install the zuul version of a thing rather than what is specified in the script | 23:46 |
| clarkb | nibalizer: to allow for integration testing | 23:46 |
| *** openstackgerrit has quit IRC | 23:46 | |
| ianw | jeblair: the "AllocationRequest for 0.0" was the bit i was missing that i think split this open | 23:46 |
| nibalizer | oh now im getting it | 23:46 |
| nibalizer | i was like 'how does not installing it make things better' | 23:47 |
| nibalizer | now im getting it | 23:47 |
| nibalizer | :D | 23:47 |
| *** jamielennox|away is now known as jamielennox | 23:47 | |
| wenlock | clarkb if you did it like my change, then you could have a module.xyz.env file per environment and just set xyz to what you want to install | 23:47 |
| ianw | jeblair: i can work on test-case, but if you already have a head-start, that's great | 23:47 |
| *** openstackgerrit has joined #openstack-infra | 23:47 | |
| jeblair | ianw: nope, i was working solely with fake servers, etc; i think you have the head start on test cases | 23:48 |
| clarkb | wenlock: ya use a conditional on a file existing and being passed in or some such | 23:48 |
| wenlock | clarkb cool, i have some full integration testing to do on that tomorrow, then i'll shoot over a change request on review.o.o | 23:49 |
| clarkb | jeblair: the thing a pid can do is skew your outputs based on (un)expected inputs. So if you ask for 1 f20 images and never get it over time it could choose to do 2, then 3 and so on | 23:49 |
| fungi | clarkb: i'm with you on the pid loop being cool just so we can have integral and derivative calculations being performed in nodepool (so i feel like the years i spent in calculus actually went to a real use) | 23:50 |
| *** tsg has joined #openstack-infra | 23:51 | |
| *** gokrokve has quit IRC | 23:51 | |
| ianw | fungi: you should build a souis-vide immersion circulator using an arduino or something. that way you can have PID and delicious steak too | 23:51 |
| openstackgerrit | A change was merged to openstack-infra/config: Adding project saml-generator https://review.openstack.org/110378 | 23:52 |
| fungi | ianw: great idea | 23:52 |
| clarkb | but but fire! | 23:52 |
| clarkb | I should do a brisket | 23:53 |
| ianw | clarkb: you still need fire at the end https://www.kickstarter.com/projects/1708738346/the-searzall :) | 23:53 |
| *** tomoe_ has joined #openstack-infra | 23:53 | |
| *** tomoe_ has joined #openstack-infra | 23:54 | |
| ianw | jeblair: would you be happy with fix & expanded test cases within the history-allocator patch itself? or were you thinking of stripping out 0 allocations separately? | 23:55 |
| *** wenlock has quit IRC | 23:55 | |
| fungi | heh, searzall.... brings back memories of family dinners with dad finishing platters on the hearth with a stand-up acetylene torch | 23:57 |
| clarkb | fungi: jeblair any concern approving 106402? | 23:57 |
| clarkb | the only major concner on my end is it "enables" the iptables persistent service which should be enabled on debian based distros anyways | 23:58 |
| clarkb | so not reall yworried but its firewalls so asking | 23:58 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!