| anteaya | do we know any six maintainers? | 00:00 |
|---|---|---|
| Alex_Gaynor | I already pinged the six maintainer | 00:00 |
| anteaya | thanks | 00:00 |
| anteaya | what timezone is that person in? | 00:00 |
| Alex_Gaynor | West coast US | 00:00 |
| anteaya | why do they release on a sunday? | 00:00 |
| Alex_Gaynor | :shrug: | 00:01 |
| mordred | Alex_Gaynor: we should start keeping a record of how many times openstack deps make releases and how many times they break things and present it next year at pycon | 00:01 |
| anteaya | it is 5pm on a Sunday west coast time, why is that the preferred release time? | 00:01 |
| mordred | Alex_Gaynor: you know, maybe during tip bof | 00:01 |
| Alex_Gaynor | That's when he has some free time I guess? | 00:01 |
| Alex_Gaynor | mordred: Pretty sure the graph is something like: 80% setuptools, 20% everything else | 00:01 |
| anteaya | interesting | 00:01 |
| mordred | Alex_Gaynor: :) | 00:01 |
| Alex_Gaynor | I feel bad for the setuptools maintainer, he inherited a project with limited testing infrastructure, and 10 years of bizzare hacks on top of it | 00:02 |
| Alex_Gaynor | if this isn't fixed in a few, I'll send a PR to pin six | 00:02 |
| *** yamahata has quit IRC | 00:04 | |
| *** adrian_otto has quit IRC | 00:14 | |
| mordred | Alex_Gaynor: yah. | 00:14 |
| *** markmcclain1 has quit IRC | 00:14 | |
| *** sarob has joined #openstack-infra | 00:15 | |
| clarkb | dont pin six | 00:16 |
| clarkb | pinning six is worst thing ever | 00:17 |
| Alex_Gaynor | noted | 00:17 |
| clarkb | it is a forward compat lib so pretty much needs to be unpinned :( | 00:17 |
| *** sarob has quit IRC | 00:19 | |
| *** matsuhashi has joined #openstack-infra | 00:21 | |
| mordred | clarkb: but it broken | 00:25 |
| clarkb | last time we pinned six it made everything break | 00:25 |
| mordred | clarkb: yah. I mean, I agree with you | 00:26 |
| clarkb | if you need to exclude a six release | 00:26 |
| mordred | clarkb: but also, it's broken | 00:26 |
| clarkb | dont pin it | 00:26 |
| mordred | oh - just adda !x.x.x yeah? | 00:26 |
| clarkb | yes | 00:26 |
| mordred | nod | 00:26 |
| clarkb | or is it != ? | 00:27 |
| clarkb | in any case do that :) | 00:27 |
| *** bearhands is now known as comstud | 00:29 | |
| mordred | clarkb: what are you doing here - aren't you supposed to be in BC? | 00:30 |
| clarkb | I am waiting for my return boat | 00:33 |
| clarkb | the museum kicked everyone out at 5 | 00:33 |
| anteaya | clarkb: did you have a good time until you got kicked out of the museum? | 00:34 |
| clarkb | yup victoria is nice | 00:34 |
| anteaya | yay | 00:35 |
| anteaya | anything memorable? | 00:35 |
| anteaya | did you see anything that caught your attention? | 00:35 |
| clarkb | also the chip and pin cards have a nice side effect of paying the bill at your table so you can let the computer figure out tip | 00:36 |
| anteaya | yes | 00:36 |
| clarkb | the parliament building tour was good | 00:36 |
| anteaya | nice | 00:36 |
| anteaya | did they talk about just the building? | 00:36 |
| clarkb | and royal bc museum is good | 00:37 |
| anteaya | or did you get a helping of canadian politics | 00:37 |
| anteaya | cool | 00:37 |
| anteaya | did you see any first nations art? | 00:37 |
| clarkb | not politics but government structure | 00:37 |
| anteaya | cool | 00:37 |
| mordred | ok. just so everyone knows - the first run of the stackalytics processor takes FOREVER | 00:38 |
| mordred | and it is spending 80% of the time trying to get info from the openstack foudation community management system | 00:38 |
| anteaya | interesting to know | 00:39 |
| mordred | YorikSar: any idea what it thinks it's going to do with that data? | 00:39 |
| clarkb | anteaya: also I learned everything costs more but food is better | 00:41 |
| clarkb | and yall do nationalistic pride like we do | 00:41 |
| mordred | ah. it seems it's for a report that exists but that is unlinked from the main page so far | 00:41 |
| mordred | http://stackalytics.com/report/members | 00:42 |
| * mordred assumes this means it's a work-in-progress report | 00:42 | |
| anteaya | clarkb: yeah, we be fairly unhumble in some respects | 00:49 |
| anteaya | the old retired folks have more time for flag waving than others | 00:49 |
| anteaya | Victoria is the retirement capital of Canada | 00:49 |
| anteaya | glad you are enjoying the food | 00:49 |
| anteaya | costs are higher since restaurant staff are paid a minimum wage | 00:50 |
| *** michchap has joined #openstack-infra | 00:50 | |
| clarkb | they are in washington and oregon too | 00:51 |
| anteaya | cool | 00:51 |
| anteaya | I am glad to hear that | 00:51 |
| anteaya | yay washington and oregon | 00:51 |
| *** nati_ueno has quit IRC | 00:53 | |
| clarkb | beacon hill park was neat too | 00:54 |
| clarkb | the herons are nesting right now | 00:54 |
| mordred | ooh. herons | 00:55 |
| *** UtahDave has joined #openstack-infra | 01:03 | |
| *** UtahDave has left #openstack-infra | 01:04 | |
| mordred | ok. http://stackalytics.openstack.org is running and populated with data. there's a 4 hour cron job | 01:04 |
| mordred | it's a little wonky, which I think is probably due to the differences in mod_wsgi and uwsgi | 01:05 |
| mordred | so we probably do want to figure out how to run in a uwsgi container | 01:05 |
| anteaya | I have never been to beacon hill | 01:09 |
| anteaya | I feel I am missing out | 01:09 |
| anteaya | yay stacklytics.o.o | 01:09 |
| clarkb | and did you know women fill the 3 most powerful positions of BC government | 01:11 |
| alkari1 | @ mordered @mordred_phone : Thank you | 01:13 |
| anteaya | clarkb: I know that the premier is female | 01:13 |
| anteaya | I didn't know women had the 3 most powerful positions though | 01:14 |
| clarkb | speaker and lt gov too apparently | 01:14 |
| anteaya | for a time we had parity in provinciap premiers | 01:14 |
| anteaya | cool | 01:14 |
| anteaya | we had 6 women and 7 men as leaders of provincial governments | 01:14 |
| anteaya | I was so proud seeing that photo | 01:14 |
| *** sarob has joined #openstack-infra | 01:15 | |
| *** nati_ueno has joined #openstack-infra | 01:19 | |
| *** sarob has quit IRC | 01:19 | |
| *** nati_ueno has quit IRC | 01:20 | |
| *** yamahata has joined #openstack-infra | 01:23 | |
| *** russellb has joined #openstack-infra | 01:27 | |
| *** amotoki has joined #openstack-infra | 01:36 | |
| *** wenlock has joined #openstack-infra | 01:36 | |
| *** wenlock has quit IRC | 01:40 | |
| *** nosnos has joined #openstack-infra | 01:43 | |
| *** yaguang has joined #openstack-infra | 01:48 | |
| *** pcrews_ has joined #openstack-infra | 01:49 | |
| *** pcrews has quit IRC | 01:52 | |
| *** yamahata has quit IRC | 01:54 | |
| *** yamahata has joined #openstack-infra | 01:54 | |
| *** mriedem has quit IRC | 02:03 | |
| *** wenlock has joined #openstack-infra | 02:10 | |
| *** sarob has joined #openstack-infra | 02:15 | |
| *** pcrews_ has quit IRC | 02:16 | |
| *** harlowja_at_home has joined #openstack-infra | 02:17 | |
| *** sarob has quit IRC | 02:19 | |
| *** julim has quit IRC | 02:26 | |
| mgagne1 | if we could have a way to permalink comments in Gerrit, that would be great. | 02:30 |
| *** mgagne1 is now known as mgagne | 02:30 | |
| *** zhiyan_ is now known as zhiyan | 02:35 | |
| mgagne | zaro: ^ =) | 02:38 |
| *** homeless has joined #openstack-infra | 02:57 | |
| *** harlowja_at_home has quit IRC | 02:59 | |
| *** gokrokve has joined #openstack-infra | 02:59 | |
| *** wenlock has quit IRC | 03:02 | |
| *** harlowja_at_home has joined #openstack-infra | 03:07 | |
| *** lbragstad has joined #openstack-infra | 03:07 | |
| *** sarob has joined #openstack-infra | 03:15 | |
| *** nosnos has quit IRC | 03:18 | |
| *** sarob has quit IRC | 03:20 | |
| *** talluri has joined #openstack-infra | 03:23 | |
| *** zz_gondoi is now known as gondoi | 03:24 | |
| *** gondoi is now known as zz_gondoi | 03:26 | |
| lifeless | where do gertty bugs belong ? | 03:33 |
| lifeless | also, I'm lost on why https://review.openstack.org/#/c/93253/ is throwing a jenkins error. | 03:34 |
| *** gokrokve has quit IRC | 03:38 | |
| *** homeless has quit IRC | 03:39 | |
| *** gokrokve has joined #openstack-infra | 03:39 | |
| *** gokrokve has quit IRC | 03:43 | |
| harlowja_at_home | lifeless, jenkins mad at u it seems | 03:44 |
| lifeless | harlowja_at_home: its mad mad mad | 03:52 |
| harlowja_at_home | :) | 03:52 |
| harlowja_at_home | like a hatter, ha | 03:57 |
| Alex_Gaynor | clarkb, anteaya, mordred : six is fixed | 04:03 |
| *** matsuhashi has quit IRC | 04:04 | |
| *** nosnos has joined #openstack-infra | 04:06 | |
| *** matsuhashi has joined #openstack-infra | 04:13 | |
| *** godp1301 has joined #openstack-infra | 04:13 | |
| *** sarob has joined #openstack-infra | 04:15 | |
| *** yfried has quit IRC | 04:15 | |
| *** sarob has quit IRC | 04:20 | |
| *** godp1301 has quit IRC | 04:33 | |
| *** praneshp has quit IRC | 04:42 | |
| *** praneshp has joined #openstack-infra | 04:42 | |
| lifeless | blah, gertty crash - antone seen | 04:48 |
| lifeless | File "/home/robertc/.virtualenvs/openstack/local/lib/python2.7/site-packages/gertty/view/diff.py", line 340, i | 04:48 |
| lifeless | diff = self.file_diffs[oldnew][path] | 04:48 |
| lifeless | KeyError: '/COMMIT_MSG' | 04:48 |
| lifeless | anyone, that is | 04:48 |
| *** yfried has joined #openstack-infra | 04:48 | |
| *** sarob has joined #openstack-infra | 04:54 | |
| *** sarob has quit IRC | 04:58 | |
| *** lbragstad has quit IRC | 05:01 | |
| *** Longgeek has joined #openstack-infra | 05:04 | |
| *** zhiyan is now known as zhiyan_ | 05:04 | |
| YorikSar | mordred: No idea. I guess it's fetching info about developers and company affilation. | 05:12 |
| *** sarob has joined #openstack-infra | 05:15 | |
| *** sarob has quit IRC | 05:19 | |
| *** gokrokve has joined #openstack-infra | 05:27 | |
| lifeless | where does one report bugs about not being able to report bugs on gertty in storyboard ? | 05:30 |
| *** adrian_otto has joined #openstack-infra | 05:32 | |
| *** gokrokve has quit IRC | 05:33 | |
| *** renlt has joined #openstack-infra | 05:34 | |
| *** che-arne has joined #openstack-infra | 05:36 | |
| *** _nadya_ has joined #openstack-infra | 05:41 | |
| flashgordon | I see no progress on the gate over the weekend | 05:46 |
| *** lcheng has joined #openstack-infra | 05:46 | |
| *** gokrokve has joined #openstack-infra | 05:46 | |
| *** flashgordon is now known as jogo | 05:46 | |
| *** _nadya_ has quit IRC | 05:48 | |
| harlowja_at_home | jogo, ya, it seems to have gotten a teenie bit better, but not much | 05:53 |
| lifeless | jogo: indeed | 05:53 |
| lifeless | jogo: I'm seeing a bug you downgraded from critical to high now showing up on rechecks | 05:53 |
| lifeless | jogo: recheck bug 1275500 | 05:54 |
| uvirtbot | Launchpad bug 1275500 in nova "Cannot reboot instance: Network filter not found: Could not find filter" [High,Confirmed] https://launchpad.net/bugs/1275500 | 05:54 |
| jogo | matt riedemann made some good progress on https://bugs.launchpad.net/tempest/+bug/1298472 | 05:56 |
| uvirtbot | Launchpad bug 1298472 in tempest "SSHTimeout in tempest scenario tests using nova-network" [Critical,In progress] | 05:56 |
| jogo | lifeless: hmm if there is no ER query for that make sure there is | 05:57 |
| *** jcoufal has joined #openstack-infra | 05:57 | |
| jogo | and that will help us better triage its real priority | 05:57 |
| lifeless | jogo: there is | 05:59 |
| lifeless | jogo: 10-15 a day started up recently | 05:59 |
| lifeless | so its not at the 80/day level | 05:59 |
| lifeless | but its not trivial either | 05:59 |
| *** basha has joined #openstack-infra | 06:01 | |
| jogo | lifeless: bumping that back up to critical seems reasoanble to me | 06:01 |
| *** aconrad_ has quit IRC | 06:01 | |
| lifeless | doing so | 06:01 |
| lifeless | ok so sphinx is driving me batty | 06:01 |
| lifeless | on my pbr spec in oslo-specs | 06:01 |
| jogo | oh haha yeah | 06:02 |
| jogo | its very finicky | 06:02 |
| lifeless | 06:02 | |
| lifeless | /home/robertc/work/oslo-specs/doc/source/specs/juno/pbr-semver.rst:202: SEVERE: Title level inconsistent: | 06:02 |
| lifeless | Assignee(s) | 06:02 |
| lifeless | Assignee(s) a | 06:02 |
| SergeyLukjanov | folks, I've been afk whole weekend << mordred, fungi, clarkb | 06:02 |
| lifeless | ----------- | 06:02 |
| lifeless | ------------- | 06:02 |
| lifeless | if I change that to | 06:02 |
| jogo | ohh that one | 06:02 |
| lifeless | it works Just Fine | 06:02 |
| lifeless | W | 06:02 |
| lifeless | T | 06:02 |
| lifeless | F | 06:02 |
| lifeless | same indent character | 06:02 |
| jogo | anyway back to the weekend for me. | 06:03 |
| jogo | have fun lifeless ;) | 06:03 |
| *** trinaths has joined #openstack-infra | 06:03 | |
| lifeless | jogo: no suggestions ? | 06:03 |
| jogo | lifeless: patch? | 06:03 |
| lifeless | jogo: boom tish | 06:04 |
| lifeless | jogo: btw want to meet up when i"m in sunnyvale? | 06:04 |
| jogo | sure, when will that be? | 06:04 |
| lifeless | see your work email or g+ :P | 06:04 |
| lifeless | 22nd through 27th | 06:04 |
| jogo | my work email won't me me login for some reason -- sorting out tomorrow | 06:05 |
| jogo | g+ whats that ? haha anyway sounds good | 06:05 |
| jogo | o/ | 06:05 |
| *** rgerganov has joined #openstack-infra | 06:06 | |
| *** Ryan_Lane has quit IRC | 06:07 | |
| *** zhiyan_ is now known as zhiyan | 06:08 | |
| *** lcheng has quit IRC | 06:08 | |
| *** rdopiera has joined #openstack-infra | 06:10 | |
| SergeyLukjanov | mordred, btw I don't know how stackalytics works internally, only high-level things | 06:10 |
| *** dizquierdo has joined #openstack-infra | 06:13 | |
| *** sarob has joined #openstack-infra | 06:15 | |
| *** lcheng has joined #openstack-infra | 06:15 | |
| *** rickerc has quit IRC | 06:16 | |
| *** sarob has quit IRC | 06:19 | |
| *** gokrokve has quit IRC | 06:24 | |
| *** gokrokve has joined #openstack-infra | 06:25 | |
| *** gokrokve has quit IRC | 06:29 | |
| *** _nadya_ has joined #openstack-infra | 06:33 | |
| krotscheck | lifeless: What’s wrong with storyboard? | 06:35 |
| krotscheck | lifeless: Ah, nevermind | 06:36 |
| *** penguinRaider has joined #openstack-infra | 06:39 | |
| *** rickerc has joined #openstack-infra | 06:39 | |
| lifeless | krotscheck: I wanted to file a bug on gertty | 06:41 |
| lifeless | krotscheck: gertty claims its using storyboard as bugtracker | 06:41 |
| lifeless | krotscheck: I tried to add a story but it appeared to do nothing | 06:41 |
| lifeless | krotscheck: I then tried to file a bug on storyboard itself, but that appeared to do nothing. | 06:41 |
| lifeless | krotscheck: then I asked here. | 06:41 |
| *** Longgeek has quit IRC | 06:46 | |
| *** davidhadas has joined #openstack-infra | 06:47 | |
| *** doude has joined #openstack-infra | 06:49 | |
| *** jlibosva has joined #openstack-infra | 06:56 | |
| *** jpich has joined #openstack-infra | 06:56 | |
| *** harlowja_at_home has quit IRC | 06:56 | |
| *** salv-orlando has joined #openstack-infra | 06:57 | |
| *** nosnos has quit IRC | 06:57 | |
| *** gokrokve has joined #openstack-infra | 06:57 | |
| *** salv-orlando has quit IRC | 06:59 | |
| krotscheck | lifeless: Yeah, I’ve already found the issue | 06:59 |
| krotscheck | Patch incoming shortly | 06:59 |
| *** che-arne_afk has joined #openstack-infra | 06:59 | |
| *** davidhadas has quit IRC | 07:00 | |
| *** salv-orlando has joined #openstack-infra | 07:00 | |
| *** davidhadas has joined #openstack-infra | 07:00 | |
| *** salv-orlando has quit IRC | 07:00 | |
| *** gokrokve has quit IRC | 07:02 | |
| *** che-arne has quit IRC | 07:03 | |
| *** skolekonov has joined #openstack-infra | 07:03 | |
| krotscheck | lifeless: I lied. I found something that’s breaking storyboard, however it’s probably not related to your ability to create a story (i.e. I was able to create a new story just now). Did you happen to capture your http request log to see what the server reported back? | 07:07 |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Revert "Name fields checked with regex" https://review.openstack.org/98701 | 07:08 |
| krotscheck | NikitaKonovalov: I was just working on that! | 07:08 |
| krotscheck | Congrats on the fast fix :) | 07:08 |
| NikitaKonovalov | I've just noticed that Storyboard is not serving stories at all | 07:09 |
| krotscheck | NikitaKonovalov: Yeah, we’ve got a bunch of stories with nonconforming characters in the titles. | 07:09 |
| lifeless | krotscheck: I did not | 07:14 |
| * krotscheck makes a note to add good error messaging in the UI.... | 07:15 | |
| openstackgerrit | A change was merged to openstack-infra/storyboard: Revert "Name fields checked with regex" https://review.openstack.org/98701 | 07:15 |
| lifeless | krotscheck: let me bring it up again | 07:15 |
| *** sarob has joined #openstack-infra | 07:15 | |
| lifeless | krotscheck: looks like filing wasn't broken - listing was | 07:15 |
| lifeless | krotscheck: so I've filed dups... | 07:16 |
| lifeless | clarkb: ^ | 07:16 |
| krotscheck | lifeless: Makes sense. That merge that just went in should have fixed the listing, give it a bit to propagate. | 07:16 |
| *** e0ne has joined #openstack-infra | 07:16 | |
| *** Longgeek has joined #openstack-infra | 07:16 | |
| lifeless | krotscheck: how do I delete https://storyboard.openstack.org/#!/story/125 ? | 07:16 |
| krotscheck | lifeless: Click on the little pencil icon next to the title, then click on “Remove this story" | 07:17 |
| lifeless | and 124 ? | 07:17 |
| lifeless | I didn't see a pencil | 07:18 |
| lifeless | is it possible that wouldn't show up if you're not logged in? | 07:18 |
| lifeless | hah | 07:18 |
| lifeless | and after deleting 125 | 07:19 |
| lifeless | its logged me out again | 07:19 |
| lifeless | thats a bit harsh | 07:19 |
| *** sarob has quit IRC | 07:19 | |
| lifeless | krotscheck: is hiding some but not all the UI for actions when logged out deliberate or oversight ? | 07:20 |
| lifeless | (the answer influences the bug I'll file :)) | 07:20 |
| lifeless | and the bugs didn't get deleted | 07:22 |
| lifeless | ok, I'll file a bug on that | 07:22 |
| lifeless | also on the speed | 07:22 |
| *** Longgeek has quit IRC | 07:22 | |
| *** ihrachyshka has joined #openstack-infra | 07:27 | |
| *** andreykurilin_ has joined #openstack-infra | 07:29 | |
| *** che-arne has joined #openstack-infra | 07:30 | |
| *** che-arne_afk has quit IRC | 07:31 | |
| *** lcheng has quit IRC | 07:32 | |
| *** tkelsey has joined #openstack-infra | 07:34 | |
| lifeless | krotscheck: filed a bunch of bugs | 07:37 |
| lifeless | krotscheck: all of them should be reproducible in dev if you put a latency applier on your local connection (and I *reallllly* encourage you to do that - it makes visibility into latency related performance issues about a 1000% easier) | 07:38 |
| lifeless | krotscheck: [that and running ssl locally too, because of the changed cache rules HTTPS has] | 07:38 |
| *** che-arne_afk has joined #openstack-infra | 07:52 | |
| *** markmc has joined #openstack-infra | 07:53 | |
| *** che-arne_afk_ has joined #openstack-infra | 07:56 | |
| *** che-arne has quit IRC | 07:56 | |
| *** pblaho has joined #openstack-infra | 07:57 | |
| *** gokrokve has joined #openstack-infra | 07:57 | |
| *** che-arne_afk has quit IRC | 07:57 | |
| *** jistr has joined #openstack-infra | 08:00 | |
| *** krotscheck has quit IRC | 08:01 | |
| *** flaper87|afk is now known as flaper87 | 08:01 | |
| *** russell_h has quit IRC | 08:01 | |
| *** gokrokve has quit IRC | 08:02 | |
| *** russell_h has joined #openstack-infra | 08:02 | |
| *** openstackgerrit has quit IRC | 08:02 | |
| *** penguinRaider has quit IRC | 08:03 | |
| *** penguinRaider has joined #openstack-infra | 08:03 | |
| *** krotscheck has joined #openstack-infra | 08:03 | |
| *** masayukig has quit IRC | 08:05 | |
| *** masayukig_ has joined #openstack-infra | 08:06 | |
| *** gilliard has joined #openstack-infra | 08:06 | |
| *** davidhadas_ has joined #openstack-infra | 08:06 | |
| *** bradm has quit IRC | 08:06 | |
| *** openstackgerrit has joined #openstack-infra | 08:07 | |
| *** davidhadas has quit IRC | 08:08 | |
| *** plomakin has quit IRC | 08:08 | |
| *** derekh_afk is now known as derekh_ | 08:09 | |
| *** praneshp has quit IRC | 08:09 | |
| *** anteaya has quit IRC | 08:10 | |
| *** slamont has quit IRC | 08:11 | |
| *** maurosr has quit IRC | 08:11 | |
| *** slamont has joined #openstack-infra | 08:13 | |
| *** russell_h has quit IRC | 08:13 | |
| *** plomakin has joined #openstack-infra | 08:13 | |
| *** maurosr has joined #openstack-infra | 08:14 | |
| *** Hunner has quit IRC | 08:14 | |
| *** sarob has joined #openstack-infra | 08:15 | |
| *** andreaf has joined #openstack-infra | 08:16 | |
| *** russell_h has joined #openstack-infra | 08:16 | |
| *** renlt has quit IRC | 08:17 | |
| *** sarob has quit IRC | 08:19 | |
| *** russell_h has quit IRC | 08:21 | |
| *** mattymo has quit IRC | 08:21 | |
| *** med_ has joined #openstack-infra | 08:22 | |
| *** med_ has quit IRC | 08:22 | |
| *** med_ has joined #openstack-infra | 08:22 | |
| *** tristanC has quit IRC | 08:23 | |
| *** mattymo has joined #openstack-infra | 08:24 | |
| *** mattymo has quit IRC | 08:24 | |
| *** russell_h has joined #openstack-infra | 08:24 | |
| *** tristanC has joined #openstack-infra | 08:24 | |
| *** mattymo has joined #openstack-infra | 08:26 | |
| *** mattymo has quit IRC | 08:26 | |
| *** med_ is now known as 1JTAAEE11 | 08:27 | |
| *** _nadya_ has quit IRC | 08:30 | |
| *** radez_g0n3 has quit IRC | 08:31 | |
| *** 1JTAAEE11 has quit IRC | 08:31 | |
| *** mattymo has joined #openstack-infra | 08:31 | |
| *** russell_h has quit IRC | 08:31 | |
| *** radez_g0n3 has joined #openstack-infra | 08:32 | |
| *** russell_h has joined #openstack-infra | 08:33 | |
| *** briancline has quit IRC | 08:35 | |
| *** proffalken has quit IRC | 08:35 | |
| *** habib has joined #openstack-infra | 08:35 | |
| *** mattymo has quit IRC | 08:36 | |
| *** proffalken has joined #openstack-infra | 08:36 | |
| *** briancline has joined #openstack-infra | 08:37 | |
| openstackgerrit | Mark McLoughlin proposed a change to openstack-infra/elastic-recheck: Match #1298472 fingerprint against console tag https://review.openstack.org/98716 | 08:38 |
| *** russell_h has quit IRC | 08:38 | |
| *** Hunner has joined #openstack-infra | 08:40 | |
| *** yamahata has quit IRC | 08:41 | |
| *** mattymo has joined #openstack-infra | 08:42 | |
| *** briancline has quit IRC | 08:42 | |
| *** russell_h has joined #openstack-infra | 08:42 | |
| *** briancline has joined #openstack-infra | 08:42 | |
| *** rlandy has joined #openstack-infra | 08:46 | |
| *** e0ne has quit IRC | 08:47 | |
| *** mattymo has quit IRC | 08:50 | |
| *** eharney has quit IRC | 08:50 | |
| *** eharney has joined #openstack-infra | 08:50 | |
| *** basha has quit IRC | 08:52 | |
| *** mattymo has joined #openstack-infra | 08:52 | |
| *** mattymo has quit IRC | 08:52 | |
| *** briancline has quit IRC | 08:54 | |
| *** mattymo has joined #openstack-infra | 08:54 | |
| *** mattymo has quit IRC | 08:54 | |
| *** che-arne_afk_ is now known as che-arne | 08:55 | |
| *** rdopiera has quit IRC | 08:56 | |
| *** mattymo has joined #openstack-infra | 08:56 | |
| *** mattymo has quit IRC | 08:56 | |
| *** mattymo has joined #openstack-infra | 08:58 | |
| *** mattymo has quit IRC | 08:58 | |
| *** briancline has joined #openstack-infra | 08:58 | |
| *** russell_h has quit IRC | 08:58 | |
| *** russell_h has joined #openstack-infra | 08:59 | |
| *** mattymo has joined #openstack-infra | 09:00 | |
| *** mattymo has quit IRC | 09:00 | |
| *** pelix has joined #openstack-infra | 09:01 | |
| *** _nadya_ has joined #openstack-infra | 09:01 | |
| *** russell_h has quit IRC | 09:02 | |
| *** IvanBerezovskiy has joined #openstack-infra | 09:02 | |
| *** pblaho has quit IRC | 09:02 | |
| *** mattymo has joined #openstack-infra | 09:02 | |
| *** mattymo has quit IRC | 09:02 | |
| *** russell_h has joined #openstack-infra | 09:04 | |
| *** mattymo has joined #openstack-infra | 09:04 | |
| *** mattymo has quit IRC | 09:04 | |
| *** briancline has quit IRC | 09:05 | |
| *** briancline has joined #openstack-infra | 09:06 | |
| *** mattymo has joined #openstack-infra | 09:06 | |
| *** mattymo has quit IRC | 09:06 | |
| *** Longgeek has joined #openstack-infra | 09:07 | |
| *** che-arne has quit IRC | 09:09 | |
| *** rgerganov_ has joined #openstack-infra | 09:10 | |
| *** Longgeek has quit IRC | 09:12 | |
| *** che-arne has joined #openstack-infra | 09:12 | |
| *** mattymo has joined #openstack-infra | 09:14 | |
| *** russell_h has quit IRC | 09:14 | |
| *** rgerganov has quit IRC | 09:14 | |
| *** russell_h has joined #openstack-infra | 09:14 | |
| *** mattymo has quit IRC | 09:14 | |
| *** sarob has joined #openstack-infra | 09:15 | |
| *** russell_h has quit IRC | 09:18 | |
| *** salv-orlando has joined #openstack-infra | 09:18 | |
| *** salv-orlando has quit IRC | 09:18 | |
| *** med_ has joined #openstack-infra | 09:18 | |
| *** med_ has quit IRC | 09:18 | |
| *** med_ has joined #openstack-infra | 09:18 | |
| *** mattymo has joined #openstack-infra | 09:19 | |
| *** roeyc has joined #openstack-infra | 09:19 | |
| *** sarob has quit IRC | 09:20 | |
| *** russell_h has joined #openstack-infra | 09:20 | |
| *** pblaho has joined #openstack-infra | 09:21 | |
| *** habib has quit IRC | 09:21 | |
| *** yolanda has joined #openstack-infra | 09:23 | |
| *** med_ has quit IRC | 09:24 | |
| *** russell_h has quit IRC | 09:26 | |
| *** salv-orlando has joined #openstack-infra | 09:27 | |
| *** gokrokve has joined #openstack-infra | 09:27 | |
| *** dosaboy_ is now known as disaboy | 09:29 | |
| *** disaboy is now known as dosaboy | 09:30 | |
| *** matsuhashi has quit IRC | 09:31 | |
| *** matsuhashi has joined #openstack-infra | 09:31 | |
| *** gokrokve has quit IRC | 09:32 | |
| *** salv-orlando has quit IRC | 09:32 | |
| *** matsuhashi has quit IRC | 09:39 | |
| *** matsuhashi has joined #openstack-infra | 09:40 | |
| *** fbo_away is now known as fbo | 09:44 | |
| *** habib has joined #openstack-infra | 09:49 | |
| *** _nadya_ has quit IRC | 09:51 | |
| *** penguinRaider has quit IRC | 09:53 | |
| *** e0ne has joined #openstack-infra | 09:55 | |
| *** russell_h has joined #openstack-infra | 10:01 | |
| *** che-arne has quit IRC | 10:02 | |
| *** che-arne has joined #openstack-infra | 10:03 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Validate Project and Project Groups names https://review.openstack.org/98728 | 10:05 |
| *** che-arne_afk has joined #openstack-infra | 10:06 | |
| *** bradm has joined #openstack-infra | 10:06 | |
| *** Hal has joined #openstack-infra | 10:08 | |
| *** Hal is now known as Guest90049 | 10:08 | |
| *** che-arne has quit IRC | 10:09 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Validate Project and Project Groups names https://review.openstack.org/98728 | 10:09 |
| *** anteaya has joined #openstack-infra | 10:10 | |
| *** sarob has joined #openstack-infra | 10:15 | |
| *** _nadya_ has joined #openstack-infra | 10:17 | |
| *** zhiyan is now known as zhiyan_ | 10:17 | |
| *** sarob has quit IRC | 10:19 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Validate Project and Project Groups names https://review.openstack.org/98728 | 10:22 |
| *** basha has joined #openstack-infra | 10:25 | |
| *** gokrokve has joined #openstack-infra | 10:27 | |
| *** e0ne has quit IRC | 10:29 | |
| *** Longgeek has joined #openstack-infra | 10:31 | |
| *** gokrokve has quit IRC | 10:31 | |
| *** che-arne has joined #openstack-infra | 10:40 | |
| *** che-arne_afk has quit IRC | 10:43 | |
| *** matsuhashi has quit IRC | 10:51 | |
| *** che-arne_afk has joined #openstack-infra | 10:55 | |
| *** matsuhashi has joined #openstack-infra | 10:58 | |
| *** che-arne has quit IRC | 10:58 | |
| *** e0ne has joined #openstack-infra | 11:00 | |
| *** Guest90049 has quit IRC | 11:01 | |
| *** e0ne has quit IRC | 11:04 | |
| *** e0ne has joined #openstack-infra | 11:06 | |
| *** habib has quit IRC | 11:13 | |
| *** sarob has joined #openstack-infra | 11:15 | |
| *** _nadya__ has joined #openstack-infra | 11:16 | |
| *** _nadya_ has quit IRC | 11:16 | |
| *** matsuhashi has quit IRC | 11:18 | |
| *** matsuhashi has joined #openstack-infra | 11:19 | |
| *** davidhadas has joined #openstack-infra | 11:20 | |
| *** sarob has quit IRC | 11:20 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Added test for Projec Groups API https://review.openstack.org/98745 | 11:22 |
| *** che-arne has joined #openstack-infra | 11:23 | |
| *** che-arne_afk has quit IRC | 11:23 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Added tests for Project Groups API https://review.openstack.org/98745 | 11:23 |
| *** davidhadas_ has quit IRC | 11:23 | |
| *** maxbit has joined #openstack-infra | 11:24 | |
| *** matsuhashi has quit IRC | 11:24 | |
| *** davidhadas has quit IRC | 11:25 | |
| *** ityaptin has quit IRC | 11:25 | |
| *** gokrokve has joined #openstack-infra | 11:27 | |
| *** gokrokve has quit IRC | 11:32 | |
| *** e0ne has quit IRC | 11:32 | |
| *** che-arne_afk has joined #openstack-infra | 11:35 | |
| *** weshay has joined #openstack-infra | 11:37 | |
| *** che-arne has quit IRC | 11:37 | |
| *** yaguang has quit IRC | 11:40 | |
| *** rlandy has quit IRC | 11:42 | |
| *** ociuhandu has joined #openstack-infra | 11:43 | |
| *** maxbit has quit IRC | 11:46 | |
| *** ociuhandu has quit IRC | 11:48 | |
| *** maxbit has joined #openstack-infra | 11:51 | |
| *** rlandy has joined #openstack-infra | 11:55 | |
| *** penguinRaider has joined #openstack-infra | 11:58 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Update method improved in db api https://review.openstack.org/98760 | 11:58 |
| *** trinaths has quit IRC | 12:01 | |
| *** dkranz has joined #openstack-infra | 12:01 | |
| openstackgerrit | Tom Fifield proposed a change to openstack-infra/config: Add a mailing list for Win The Enterprise WG https://review.openstack.org/98762 | 12:01 |
| *** dprince has joined #openstack-infra | 12:02 | |
| *** basha has quit IRC | 12:03 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Added tests for Project Groups API https://review.openstack.org/98745 | 12:03 |
| *** andreykurilin_ has quit IRC | 12:05 | |
| *** eharney has quit IRC | 12:06 | |
| *** mwagner_lap has quit IRC | 12:08 | |
| *** adalbas has joined #openstack-infra | 12:11 | |
| *** _nadya__ has quit IRC | 12:12 | |
| *** lcostantino has joined #openstack-infra | 12:12 | |
| *** _nadya_ has joined #openstack-infra | 12:14 | |
| *** sarob has joined #openstack-infra | 12:15 | |
| *** che-arne has joined #openstack-infra | 12:19 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: update 1275500 to match more broadly https://review.openstack.org/98767 | 12:19 |
| *** sarob has quit IRC | 12:20 | |
| *** che-arne_afk has quit IRC | 12:20 | |
| *** rfolco has joined #openstack-infra | 12:20 | |
| *** _nadya_ has quit IRC | 12:23 | |
| *** ArxCruz has joined #openstack-infra | 12:24 | |
| *** aysyd has joined #openstack-infra | 12:24 | |
| *** _nadya_ has joined #openstack-infra | 12:25 | |
| *** Guest90049 has joined #openstack-infra | 12:26 | |
| *** gokrokve has joined #openstack-infra | 12:27 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Match #1298472 fingerprint against console tag https://review.openstack.org/98716 | 12:27 |
| *** dkliban_afk is now known as dkliban | 12:28 | |
| *** habib has joined #openstack-infra | 12:30 | |
| *** talluri has quit IRC | 12:30 | |
| *** talluri has joined #openstack-infra | 12:31 | |
| *** gokrokve has quit IRC | 12:31 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: remove 1285908 query https://review.openstack.org/98774 | 12:32 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: update 1275500 to match more broadly https://review.openstack.org/98767 | 12:33 |
| *** che-arne_afk has joined #openstack-infra | 12:34 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add acl to create branches on powervc-driver https://review.openstack.org/97726 | 12:35 |
| *** niska has quit IRC | 12:35 | |
| *** talluri has quit IRC | 12:35 | |
| openstackgerrit | A change was merged to openstack-infra/config: Finally add osprofiler to stackforge https://review.openstack.org/97778 | 12:36 |
| *** eharney has joined #openstack-infra | 12:37 | |
| *** che-arne has quit IRC | 12:38 | |
| *** ildikov_ has quit IRC | 12:38 | |
| *** che-arne has joined #openstack-infra | 12:40 | |
| *** niska has joined #openstack-infra | 12:41 | |
| *** lcheng has joined #openstack-infra | 12:41 | |
| *** che-arne_afk has quit IRC | 12:43 | |
| fungi | anteaya: i didn't mean to imply i'd be able to graph that (i don't know who voted and who didn't but as an election official perhaps civs will tell you?), just that i could get you the raw commit counts per contributor if you have the rest of the data you need to do the comparison | 12:44 |
| *** pdmars has joined #openstack-infra | 12:45 | |
| *** Guest90049 has quit IRC | 12:46 | |
| *** Hal has joined #openstack-infra | 12:46 | |
| *** Hal is now known as Guest54241 | 12:46 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: remove 1285908 query https://review.openstack.org/98774 | 12:47 |
| *** andreykurilin_ has joined #openstack-infra | 12:47 | |
| *** annegentle has joined #openstack-infra | 12:49 | |
| Kiall | fungi: could I convince you to do another review for the designate devstack gate? https://review.openstack.org/#/c/98584/ - I've realized exercises are disabled by default, so I'd like to be able to run just the designate ones via the post_test_hook until we get some tempest plugins in place (Which I can then copy the solum pattern of installing and running them via post_test_hook) | 12:50 |
| *** che-arne_afk has joined #openstack-infra | 12:50 | |
| *** ociuhandu has joined #openstack-infra | 12:52 | |
| *** signed8bit has joined #openstack-infra | 12:52 | |
| *** smarcet has joined #openstack-infra | 12:53 | |
| *** che-arne has quit IRC | 12:53 | |
| *** habib_ has joined #openstack-infra | 12:53 | |
| *** habib_ has quit IRC | 12:53 | |
| *** habib_ has joined #openstack-infra | 12:54 | |
| *** basha has joined #openstack-infra | 12:54 | |
| *** prad has joined #openstack-infra | 12:55 | |
| *** habib has quit IRC | 12:56 | |
| *** adrian_otto has quit IRC | 12:58 | |
| *** Guest54241 has quit IRC | 12:58 | |
| *** Hal_ has joined #openstack-infra | 12:58 | |
| Kiall | fungi: thanks :) | 12:59 |
| *** miqui has joined #openstack-infra | 13:00 | |
| *** julim has joined #openstack-infra | 13:01 | |
| *** mfer has joined #openstack-infra | 13:02 | |
| openstackgerrit | A change was merged to openstack-infra/config: Move designate pre/post dsvm hooks to the designate repo https://review.openstack.org/98584 | 13:05 |
| *** basha has quit IRC | 13:06 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/infra-specs: StoryBoard search spec https://review.openstack.org/97254 | 13:06 |
| *** vhoward has joined #openstack-infra | 13:12 | |
| *** che-arne has joined #openstack-infra | 13:13 | |
| *** liyuezhou has joined #openstack-infra | 13:14 | |
| *** liyuezhou has quit IRC | 13:14 | |
| *** Samolo_ has joined #openstack-infra | 13:14 | |
| *** mriedem has joined #openstack-infra | 13:14 | |
| *** sarob has joined #openstack-infra | 13:15 | |
| *** che-arne_afk has quit IRC | 13:17 | |
| sdague | fungi: can we get a promote on this - https://review.openstack.org/#/c/98632/ | 13:17 |
| *** jaypipes has joined #openstack-infra | 13:19 | |
| mestery | Morning folks! | 13:19 |
| *** sarob has quit IRC | 13:19 | |
| *** che-arne_afk has joined #openstack-infra | 13:20 | |
| *** che-arne has quit IRC | 13:21 | |
| phschwartz | morning infra. Finally pain free and hoping to get some work done :) | 13:22 |
| *** palar has joined #openstack-infra | 13:24 | |
| *** lcheng has quit IRC | 13:27 | |
| *** gokrokve has joined #openstack-infra | 13:27 | |
| *** jaypipes has quit IRC | 13:28 | |
| *** jaypipes has joined #openstack-infra | 13:29 | |
| *** palar has quit IRC | 13:29 | |
| sdague | fungi / SergeyLukjanov either of you guys have time for a promote? | 13:30 |
| *** palar has joined #openstack-infra | 13:30 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard-webclient: Added field restrictions and error messages to project forms https://review.openstack.org/95873 | 13:30 |
| *** smarcet has quit IRC | 13:30 | |
| *** godp1301 has joined #openstack-infra | 13:31 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Validate Project and Project Groups names https://review.openstack.org/98728 | 13:32 |
| *** otherwiseguy has joined #openstack-infra | 13:32 | |
| *** gokrokve has quit IRC | 13:32 | |
| *** signed8bit has quit IRC | 13:32 | |
| *** signed8bit has joined #openstack-infra | 13:33 | |
| *** lcheng has joined #openstack-infra | 13:34 | |
| *** jaypipes has quit IRC | 13:34 | |
| *** basha has joined #openstack-infra | 13:34 | |
| *** jaypipes has joined #openstack-infra | 13:35 | |
| *** jaypipes has quit IRC | 13:36 | |
| *** jaypipes has joined #openstack-infra | 13:38 | |
| anteaya | fungi: ah okay, well it turns out it is academic anyway since the system we use simply tells me who is eligble to vote, I have access to zero data about who actually took the time to cast a vote | 13:38 |
| *** zz_gondoi is now known as gondoi | 13:39 | |
| anteaya | Alex_Gaynor: yay, thanks for helping six to be fixed | 13:39 |
| Alex_Gaynor | anteaya: Let's not give me too much credit, all I did was file an issue and complain on IRC :-) | 13:40 |
| *** pcrews_ has joined #openstack-infra | 13:40 | |
| anteaya | Alex_Gaynor: I give you credit for that | 13:40 |
| anteaya | Alex_Gaynor: thank you | 13:40 |
| Kiall | sdague: re the designate gate API call - I realized on Saturday (after sending the email on Friday) that the devstack-gate doesn't run the exercises by default anymore. fungi just landed a change to the gate job that will let me run them. The 1 API call you saw was a call to do some initial config, the output is buried in the middle of devstacklog.txt.gz @ http://bit.ly/Uo8TVQ .. the actual exercises should be running once the series of | 13:41 |
| Kiall | designate devstack patches I made over the weekend land. | 13:41 |
| *** basha has quit IRC | 13:42 | |
| *** salv-orlando has joined #openstack-infra | 13:42 | |
| sdague | Kiall: ok, cool | 13:42 |
| *** che-arne has joined #openstack-infra | 13:44 | |
| Kiall | (I *only* have 13 patches pending to our DevStack plugin - lots of cleanups + additions + updates to match current devstack patterns ;)) | 13:44 |
| *** smarcet has joined #openstack-infra | 13:45 | |
| *** jaypipes has quit IRC | 13:45 | |
| bookwar | hi all, Is it possible for third-party voting CI (fuel-ci) to become a required CI for corresponding projects(stackforge/fuel-*)? I'd like to disable merging the code if it doesn't have +1 from CI | 13:45 |
| sdague | yeh, so I'm not actually looking at patches that aren't gate related until the failure rate drops | 13:45 |
| *** jaypipes has joined #openstack-infra | 13:46 | |
| *** che-arne_afk has quit IRC | 13:47 | |
| *** jaypipes has quit IRC | 13:49 | |
| *** homeless has joined #openstack-infra | 13:49 | |
| *** jaypipes has joined #openstack-infra | 13:49 | |
| anteaya | bookwar: that is not possible | 13:50 |
| *** freyes has joined #openstack-infra | 13:50 | |
| *** basha has joined #openstack-infra | 13:50 | |
| anteaya | bookwar: code merges based on the gate | 13:50 |
| anteaya | bookwar: we aren't going to set a precident for code only merging based on feedback of a third party ci | 13:51 |
| *** basha has quit IRC | 13:51 | |
| *** e0ne has joined #openstack-infra | 13:51 | |
| anteaya | if you don't want the code to merge, but the tests in teh gate | 13:51 |
| anteaya | s/but/put | 13:51 |
| bookwar | anteaya: ok, thanks | 13:52 |
| anteaya | anytime | 13:52 |
| *** Longgeek has quit IRC | 13:55 | |
| *** Longgeek has joined #openstack-infra | 13:55 | |
| fungi | sdague: promoting 98632 now | 13:55 |
| sdague | fungi: thanks | 13:56 |
| fungi | sdague: looks like it's 4 minutes from getting enqueued, so i'm going to wait for it to end up in the gate rather than manually enqueuing | 13:57 |
| fungi | and then i'll send it to the front of the line | 13:57 |
| *** habib_ has quit IRC | 13:57 | |
| sdague | fungi: sounds good | 13:57 |
| sdague | yeh, last test is just uploading artifacts now | 13:58 |
| *** basha has joined #openstack-infra | 13:58 | |
| fungi | sdague: should we add any others behind it? (98515,2, 97757,1) | 13:58 |
| sdague | fungi: it's in | 13:58 |
| sdague | fungi: sure | 13:59 |
| *** liyuezhou has joined #openstack-infra | 13:59 | |
| openstackgerrit | Sergey Murashov proposed a change to openstack-infra/config: Make mistral-dsvm jobs voting https://review.openstack.org/98799 | 13:59 |
| *** gokrokve has joined #openstack-infra | 14:00 | |
| *** mwagner_lap has joined #openstack-infra | 14:00 | |
| fungi | done. looks like there was a failure at the head of the gate in the last few minutes, so no real loss there | 14:00 |
| sdague | the fact that py34 tox doesn't work in ubuntu makes me kind of sad | 14:00 |
| sdague | means I can't actually iterate on any python 3 fixes | 14:00 |
| fungi | sdague: known bug? need newer tox? | 14:00 |
| *** habib_ has joined #openstack-infra | 14:01 | |
| sdague | fungi: honestly, I don't know | 14:01 |
| *** salv-orlando has quit IRC | 14:01 | |
| *** beekneemech is now known as bnemec | 14:02 | |
| fungi | sdague: i see some recentish bugs (in the past ~8 months) about virtualenv support for 3.4 | 14:02 |
| openstackgerrit | Philip Marc Schwartz proposed a change to openstack-infra/config: Creation of vinz project in the openstack-infra scheme. https://review.openstack.org/93953 | 14:03 |
| fungi | sdague: if you're using our posargs methodology then you're stuck running tox 1.6 (at least until 1.8 is available) which bundles a releatively older virtualenv, so might not have 3.6 fixes | 14:03 |
| fungi | er, 3.4 fixes | 14:04 |
| *** lcheng has quit IRC | 14:04 | |
| sdague | yeh, it's just frustrating | 14:04 |
| *** rwsu has joined #openstack-infra | 14:04 | |
| sdague | http://paste.openstack.org/show/83416/ | 14:05 |
| phschwartz | fungi: anteaya: Mind taking a look at https://review.openstack.org/#/c/93953/ to make sure nothing is missing. | 14:05 |
| fungi | sdague: if so, that's probably going to delay our ability to ditch the py3k-precise nodepool node type | 14:05 |
| anteaya | phschwartz: morning, it is on my review list | 14:05 |
| fungi | here's hoping holger releases 1.8 soonsoonsoon | 14:05 |
| anteaya | top of my todo list today is sdague's state of the gate email and anything I can do to help | 14:06 |
| phschwartz | anteaya: ty, finally got back to it after my back injury now that I can sit in a chair for extended periods of time. | 14:06 |
| anteaya | a healthy gate makes for a better week | 14:06 |
| anteaya | phschwartz: I'm sorry to hear about your back, what happened? | 14:06 |
| *** jgrimm has joined #openstack-infra | 14:06 | |
| phschwartz | anteaya: slight bulging disc between L4 and L5 that I got from hitting rocks under the surf when I dove into the water at the beach to grab a 2 year old that went under and didn't come back up. | 14:07 |
| sdague | fungi: hmmm... if I purge out virtualenv from pip and use the ubuntu packaged one this seems like it's working now | 14:07 |
| anteaya | did you get the 2 year old? | 14:07 |
| *** lcheng has joined #openstack-infra | 14:07 | |
| *** salv-orlando has joined #openstack-infra | 14:08 | |
| phschwartz | anteaya: yeah. I got him and he is ok other then getting really scared that day and swallowing some water. | 14:08 |
| *** liyuezhou has quit IRC | 14:08 | |
| anteaya | phschwartz: glad to hear you saved him | 14:08 |
| phschwartz | anteaya: it was worth the back injury to make sure a child was ok | 14:09 |
| anteaya | phschwartz: I hear that | 14:09 |
| anteaya | phschwartz: well done | 14:09 |
| anteaya | phschwartz: well I hope stretching and yoga and physiotherapy are in your long term plans | 14:09 |
| anteaya | phschwartz: and yes, I will review your patch | 14:09 |
| anteaya | phschwartz: welcome back | 14:09 |
| sdague | oh, that just gets me to an obscure testr failure | 14:10 |
| phschwartz | anteaya: they are, and ty | 14:10 |
| anteaya | :D | 14:10 |
| *** _nadya_ has quit IRC | 14:10 | |
| fungi | sdague: it's possible you could upgrade to latest virtualenv in the tox environment and then reuse it to see if there's a recently fixed virtualenv bug blocking you | 14:10 |
| fungi | or it could be that testr also has 3.4 issues ;() | 14:11 |
| *** ildikov has joined #openstack-infra | 14:11 | |
| sdague | oh - https://bugs.launchpad.net/testrepository/+bug/1229445 | 14:11 |
| uvirtbot | Launchpad bug 1229445 in testrepository "db type could not be determined (dup-of: 1212909)" [High,Triaged] | 14:11 |
| uvirtbot | Launchpad bug 1212909 in testrepository "ImportError _bsddb with deadsnakes python2.6 anydbm" [Critical,Triaged] | 14:11 |
| sdague | which means you can never have a unit test env that tests both 2 and 3, because if 2 runs first it creates testr db files that 3 can't read | 14:12 |
| anteaya | fungi: I'm just thinking about stats, would it be worthwhile to do a pie graph just showing how many of 1510 contributors have 1 commit and how many have more than 1? | 14:12 |
| *** _nadya_ has joined #openstack-infra | 14:13 | |
| *** rfolco has quit IRC | 14:14 | |
| *** rfolco has joined #openstack-infra | 14:14 | |
| fungi | anteaya: what we could probably do is find the upper bound and make bands with the top x%,y%,z%,whatever contributor groupings and then count the percentage of each of those who voted | 14:14 |
| *** mbacchi has joined #openstack-infra | 14:15 | |
| *** sarob has joined #openstack-infra | 14:15 | |
| fungi | and note the number of commits at each group boundary in case we later want to consider a minimum commit cut-off or something | 14:15 |
| anteaya | okay, let's do that | 14:15 |
| *** rfolco has quit IRC | 14:15 | |
| anteaya | we can have the data as a point of conversation | 14:16 |
| *** rfolco has joined #openstack-infra | 14:16 | |
| anteaya | there are various interpretations of what would be involved to change the commit cut-off from anything other than one | 14:16 |
| fungi | i can get you the lists of voters in each band (maybe we do bands of 10% increments) | 14:16 |
| anteaya | which is why I changed directions and went with increasing participation | 14:16 |
| fungi | doing grouping is crucial however, since exposing a commit count+vote would actually destroy voter anonymity | 14:17 |
| anteaya | what would happen if we did 1 commit, 2 commits, 5 commits, 8 commits and then bands of 10? | 14:17 |
| *** trinaths has joined #openstack-infra | 14:17 | |
| anteaya | since it is the difference in the lower numbers that I really would like to see | 14:17 |
| fungi | once we have the raw correlation (which we can't publish for privacy/anonymity reasons) we can play with slicing and dicing it however we want i guess | 14:18 |
| anteaya | we don't have any data on who voted | 14:18 |
| openstackgerrit | David Johansen proposed a change to openstack-infra/jenkins-job-builder: Adding second failure option to the email-ext module https://review.openstack.org/98803 | 14:18 |
| anteaya | the system we use doesn't give me that data | 14:18 |
| *** mbacchi has quit IRC | 14:18 | |
| anteaya | I only have the voters list I fed in, so eligible to vote | 14:18 |
| fungi | anteaya: oh, civs won't tell you whether or not a particular member of the electorate voted? in that case we can't really put this analysis together | 14:18 |
| anteaya | correct, I have no idea who voted | 14:19 |
| *** vhoward has left #openstack-infra | 14:19 | |
| anteaya | I just wanted to see from the voters list, what percentage of people had 1 commit, or 2 commits, or 5 commits | 14:19 |
| *** rfolco has quit IRC | 14:19 | |
| anteaya | but I still won't know who voted and who didn't | 14:19 |
| *** sarob has quit IRC | 14:19 | |
| *** rfolco has joined #openstack-infra | 14:20 | |
| anteaya | we are guessing that people with more commits are more likely to cast a ballot | 14:20 |
| *** rfolco has quit IRC | 14:20 | |
| *** e0ne has quit IRC | 14:20 | |
| anteaya | but we have no data on that, nor will we have | 14:20 |
| *** rfolco has joined #openstack-infra | 14:20 | |
| *** Alex_Gaynor has quit IRC | 14:20 | |
| fungi | ahh, maybe. in that case stackalytics probably already has a graph for committer % breakdown (which will be ~=voters) | 14:21 |
| *** Alex_Gaynor has joined #openstack-infra | 14:21 | |
| *** [1]trinaths has joined #openstack-infra | 14:21 | |
| *** alkari1 has quit IRC | 14:21 | |
| anteaya | hmmmm, okay I will look | 14:21 |
| fungi | yeah, if there's no actual correlation to votes cast we need to worry about, then there's plenty of raw data and analysis already out there about our contributor base | 14:22 |
| *** mbacchi has joined #openstack-infra | 14:22 | |
| *** yolanda has quit IRC | 14:22 | |
| *** jcoufal has quit IRC | 14:24 | |
| *** trinaths has quit IRC | 14:24 | |
| *** [1]trinaths is now known as trinaths | 14:24 | |
| *** spiffxp_ has joined #openstack-infra | 14:25 | |
| anteaya | there is no actual graph, but for icehouse stackalytics tracks 1202 contributors, 344 of which have 1 commit so 344/1202 is 28.6 percent | 14:25 |
| anteaya | so if we assume that most of that 28.6 percent are not voting in the tc election, that is a large number | 14:26 |
| *** SlickN1k has joined #openstack-infra | 14:26 | |
| *** pdmars has quit IRC | 14:26 | |
| *** wenlock has joined #openstack-infra | 14:26 | |
| sdague | anteaya: yeh, I think that was the idea. Assume that if you had a single commit accepted the likelyhood of voting was low | 14:27 |
| *** pdmars has joined #openstack-infra | 14:27 | |
| anteaya | 318 commiters with 1 commit in havana, out of 1051 tracked contributors = 30.2% for havana | 14:28 |
| anteaya | just icehouse and havana voted last election | 14:29 |
| anteaya | but I am going to look at the other releases to see if we have been sitting around 30% single contribution perencentage | 14:29 |
| *** jistr has quit IRC | 14:29 | |
| *** rgerganov_ has quit IRC | 14:30 | |
| *** homeless has quit IRC | 14:31 | |
| *** jistr has joined #openstack-infra | 14:32 | |
| *** radez_g0n3 is now known as radez | 14:34 | |
| *** Alexei_987 has joined #openstack-infra | 14:34 | |
| *** dims has joined #openstack-infra | 14:36 | |
| *** salv-orlando has quit IRC | 14:39 | |
| openstackgerrit | Valeriy Ponomaryov proposed a change to openstack-infra/config: Added 'docs' job for manila project https://review.openstack.org/98465 | 14:42 |
| *** salv-orlando has joined #openstack-infra | 14:46 | |
| openstackgerrit | Philip Marc Schwartz proposed a change to openstack-infra/config: Creation of vinz project in the openstack-infra scheme. https://review.openstack.org/93953 | 14:46 |
| *** roeyc has quit IRC | 14:47 | |
| anteaya | I added my numbers to the bottom of this etherpad: https://etherpad.openstack.org/p/electoral-discussion-May-2014-numbers | 14:47 |
| anteaya | the thing I was wondering as I did this, how many commiters made 1 commit in a release and made 1 or more than 1 in a following release | 14:47 |
| fungi | anteaya: that gets into relatively complex data mining to trend individual contributor growth patterns in aggregate. not my field ;) | 14:48 |
| fungi | i know some people who know some people though... | 14:48 |
| *** thedodd has joined #openstack-infra | 14:50 | |
| anteaya | ha ha ha | 14:50 |
| anteaya | more than I want to get into on a Monday morning before I have had food | 14:50 |
| anteaya | or even after food | 14:51 |
| fungi | yeah, this is when you need some friends with degrees in social statistics | 14:51 |
| anteaya | so I'm not sure if I have provided clarity here, or muddied the waters | 14:51 |
| anteaya | I don't think I have any friends like that | 14:51 |
| anteaya | I'm going to have to get out of the house and meet some people | 14:52 |
| *** rkukura has quit IRC | 14:52 | |
| *** _nadya_ has quit IRC | 14:52 | |
| *** mbacchi has quit IRC | 14:52 | |
| *** otherwiseguy has quit IRC | 14:52 | |
| *** gokrokve has quit IRC | 14:53 | |
| *** mbacchi has joined #openstack-infra | 14:53 | |
| fungi | i used to work at a company (in the beforetime, in the long-long-ago) which analyzed anonymized chains of web browsing patterns out of packet captures at internet service providers. there's a whole science to that sort of analysis | 14:53 |
| *** otherwiseguy has joined #openstack-infra | 14:53 | |
| *** gokrokve has joined #openstack-infra | 14:53 | |
| *** e0ne has joined #openstack-infra | 14:53 | |
| anteaya | wow | 14:53 |
| anteaya | I don't what to know about the patterns | 14:54 |
| anteaya | s/what/want | 14:54 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/devstack-gate: do not override ceilometer default polling interval https://review.openstack.org/98815 | 14:54 |
| fungi | right. we had a department full of phd statisticians and social scientists for it | 14:54 |
| *** ihrachyshka_ has joined #openstack-infra | 14:54 | |
| *** ihrachyshka_ has quit IRC | 14:54 | |
| *** lbragstad has joined #openstack-infra | 14:54 | |
| phschwartz | Is salt actually being used in infra right now or is that still the basis of having everything ready for a time when salt might be used? | 14:54 |
| anteaya | bet you they were fun at parties | 14:54 |
| anteaya | phschwartz: salt is being removed | 14:55 |
| anteaya | phschwartz: we are moving to ansible | 14:55 |
| anteaya | mordred has some patches up | 14:55 |
| anteaya | I am sad about it | 14:55 |
| fungi | phschwartz: we never got very far into using it for anything other than running commands in aggregate across static jenkins slaves, and then got rid of most of our static jenkins slaves | 14:55 |
| phschwartz | mordred: do you have links to your patches where you moved to ansible. I want to take a look to see if I can help at all with the work I have done for our mirror of infra internal at Rax | 14:56 |
| fungi | phschwartz: however mordred has recently also been playing with ansible for remote execution, so i suggested he talk to you once you were back | 14:56 |
| anteaya | ArxCruz: are you around today? | 14:56 |
| fungi | ha! | 14:56 |
| fungi | phschwartz: you read my mind ;) | 14:56 |
| phschwartz | fungi: :) | 14:56 |
| *** ihrachyshka has quit IRC | 14:57 | |
| phschwartz | I have playbooks that use the ansible rax modules to replace launch_node.py and for automated creation of the puppetmaster so I defn am interested in where he was going with it. | 14:57 |
| *** gokrokve has quit IRC | 14:57 | |
| anteaya | phschwartz: https://review.openstack.org/#/q/status:open+project:openstack-infra/config+branch:master+topic:ansible,n,z | 14:58 |
| *** jistr has quit IRC | 15:00 | |
| phschwartz | another question. Any chance infra is interested in an update that makes domain name being used configurable instead of hard coded? I did the work to create our mirror of infra and it should be adaptable upstream | 15:00 |
| *** w00dy has joined #openstack-infra | 15:00 | |
| *** jistr has joined #openstack-infra | 15:00 | |
| *** yfried has quit IRC | 15:01 | |
| *** w00dy is now known as Guest61298 | 15:01 | |
| *** habib__ has joined #openstack-infra | 15:01 | |
| *** otherwiseguy has quit IRC | 15:01 | |
| *** otherwiseguy has joined #openstack-infra | 15:01 | |
| *** yaguang has joined #openstack-infra | 15:03 | |
| *** sulochan has quit IRC | 15:03 | |
| *** habib_ has quit IRC | 15:04 | |
| fungi | phschwartz: sounds like that might be something which could be easily abstracted out in site.pp? though there are several large puppet manifest refactoring efforts underway one of which might already be doing that | 15:04 |
| * fungi needs to get caught up on reviews | 15:05 | |
| *** jasond` has joined #openstack-infra | 15:05 | |
| *** ayoung has quit IRC | 15:05 | |
| phschwartz | fungi: that is exactly how I did it. in site.pp. I will wait to see what else is going on. I want to finish reviewing these reviews from mordred first | 15:05 |
| *** sulochan has joined #openstack-infra | 15:05 | |
| *** habib__ has quit IRC | 15:05 | |
| *** habib__ has joined #openstack-infra | 15:06 | |
| *** lcheng has quit IRC | 15:06 | |
| *** pdmars_ has joined #openstack-infra | 15:06 | |
| *** adrian_otto has joined #openstack-infra | 15:07 | |
| *** habib__ has quit IRC | 15:07 | |
| *** mriedem has quit IRC | 15:07 | |
| fungi | phschwartz: in fact, if you want to go hunting in openstack-infra/config changes for ongoing puppet improvements which need reviewing, and help pick them apart for bugs, that would be a huge help | 15:08 |
| *** jasond has quit IRC | 15:09 | |
| phschwartz | fungi: I will do that today. I promised you guys I would get more involved so might as well live up to that promise :) | 15:09 |
| *** jaypipes has quit IRC | 15:09 | |
| *** basha has quit IRC | 15:09 | |
| fungi | phschwartz: there is no better way to get involved than to review, so thanks! | 15:10 |
| *** pdmars has quit IRC | 15:10 | |
| *** che-arne_afk has joined #openstack-infra | 15:10 | |
| *** godp13011 has joined #openstack-infra | 15:11 | |
| *** mriedem has joined #openstack-infra | 15:11 | |
| *** che-arne has quit IRC | 15:13 | |
| *** virmitio has joined #openstack-infra | 15:14 | |
| *** arnaud__ has joined #openstack-infra | 15:15 | |
| *** sarob has joined #openstack-infra | 15:15 | |
| *** pdmars_ has quit IRC | 15:16 | |
| *** james_li has joined #openstack-infra | 15:16 | |
| *** dangers_away is now known as dangers | 15:16 | |
| *** lcheng has joined #openstack-infra | 15:18 | |
| *** morganfainberg_Z is now known as morganfainberg | 15:18 | |
| *** che-arne has joined #openstack-infra | 15:19 | |
| *** sarob has quit IRC | 15:19 | |
| *** arnaud__ has quit IRC | 15:20 | |
| *** jaypipes has joined #openstack-infra | 15:21 | |
| *** che-arne_afk has quit IRC | 15:22 | |
| *** homeless has joined #openstack-infra | 15:23 | |
| *** julim has quit IRC | 15:24 | |
| *** terryw has joined #openstack-infra | 15:24 | |
| Samolo_ | è | 15:24 |
| *** otherwiseguy has quit IRC | 15:25 | |
| *** UtahDave has joined #openstack-infra | 15:25 | |
| phschwartz | fungi: anteaya: ok, I am blanking or just not been around enough recently. Can one of you point me to what I might have screwed up. http://logs.openstack.org/53/93953/6/check/gate-config-layout/fefbbe7/console.html | 15:26 |
| *** terryw is now known as otherwiseguy | 15:26 | |
| dhellmann | mordred: how did you manage to vote "verified" on https://review.openstack.org/#/c/97872/ ? | 15:26 |
| *** rkukura has joined #openstack-infra | 15:27 | |
| sdague | dhellmann: extra special powers | 15:27 |
| dhellmann | sdague: I thought maybe he had turned into a bot. | 15:27 |
| *** gyee has joined #openstack-infra | 15:27 | |
| *** penguinRaider has quit IRC | 15:27 | |
| *** slamont has quit IRC | 15:27 | |
| fungi | phschwartz: "Job vinz-docs [...] vinz-pypi-upload not defined" so probably something's off-kilter with your job templates... looking at those now | 15:27 |
| *** freyes has quit IRC | 15:28 | |
| anteaya | phschwartz: py33 isn't in python-jobs | 15:29 |
| *** jgrimm has quit IRC | 15:29 | |
| anteaya | you need python3-jobs (I think) to call py33 jobs | 15:29 |
| *** rlandy_ has joined #openstack-infra | 15:29 | |
| fungi | phschwartz: oh, well for starters {name}-pypi-upload isn't a job template any longer... now there are {name}-pypi-wheel-upload and {name}-pypi-both-upload | 15:29 |
| fungi | (the latter of which uploads both wheels and tarballs) | 15:30 |
| mordred | dhellmann: I used special cow powers | 15:30 |
| phschwartz | ah, I see the issue. The first failure I had was because the code was outdated. I rebased and didn't notice that the jobs had changed | 15:30 |
| mordred | dhellmann: (actually, that was an oops, sorry about that) | 15:31 |
| anteaya | mordred: stay in your lane | 15:31 |
| fungi | mordred: super cow powers? (¡supercow al rescate!) | 15:31 |
| mordred | phschwartz: hey! | 15:32 |
| phschwartz | fungi: let me fix those issues and also mirror the puppet-storyboard additions also | 15:32 |
| fungi | phschwartz: sounds like a good idea | 15:32 |
| *** rlandy has quit IRC | 15:32 | |
| *** russell_h has quit IRC | 15:32 | |
| *** russell_h has joined #openstack-infra | 15:32 | |
| phschwartz | mordred: I took a look at your reviews for ansible. Like what is there :) | 15:33 |
| phschwartz | mordred: I am going to adapt my stuff to use it instead of bothering with more changes to it. | 15:33 |
| phschwartz | mordred: I did have one question on one review that I posted as a comment in it though | 15:33 |
| mordred | phschwartz: awesome! I'm very pleased with it so far - in fact, I think I'm going to submit a talk somewhere about "ansible+puppet" | 15:33 |
| * mordred goes to look | 15:33 | |
| *** jpich has quit IRC | 15:34 | |
| phschwartz | mordred: That is how we run all of our infra here at Rax. | 15:34 |
| phschwartz | ansible + puppet in masterless mode. | 15:34 |
| fungi | ansipet... puppible... | 15:34 |
| mordred | phschwartz: ++ | 15:34 |
| mordred | phschwartz: I don't see the question... | 15:35 |
| phschwartz | mordred: it is on https://review.openstack.org/96362 | 15:35 |
| *** basha has joined #openstack-infra | 15:35 | |
| *** adrian_otto has quit IRC | 15:36 | |
| mordred | thanks | 15:36 |
| *** skolekonov has quit IRC | 15:36 | |
| phschwartz | mordred: I have been trying to figure the best way to work out how infra uses puppet and how we have things internal here. I have been having trouble with puppetmaster as I am just not familiar with how it does key management | 15:36 |
| mordred | phschwartz: ah - so my idea is that we'd land the one before this one and let it run to remove things, and then later we'd land that patch | 15:37 |
| phschwartz | mordred: That is what I thought which is why I wanted to ask and make sure. | 15:37 |
| mordred | phschwartz: so, the main thing with puppet master is that in a manual world it expects that the first time a node runs puppet, it will contact the puppetmaster and this will create a cert signing request on the puppetmaster | 15:37 |
| mordred | and then on the puppetmaster you're supposed to go and say "oh, look, the mail server wants to join" | 15:38 |
| mordred | this is, of course, stupid | 15:38 |
| mordred | so in launch_node, we run commands on the puppetmaster to pre-create a key for a given hostname, then we copy that key on to the host when we create it | 15:38 |
| mordred | so that when it talks to the master it's already been pre-signed on the master | 15:38 |
| *** dizquierdo has quit IRC | 15:38 | |
| phschwartz | ok, I see what your doing and that is what got me confused. I should have traced through it more. | 15:39 |
| *** maxbit_ has joined #openstack-infra | 15:39 | |
| fungi | this has one crypto down-side, which is that anyone who gets their hands on a copy of the keys from the puppet master can now impersonate any server it knows about. in practice this is basically a non-issue however | 15:39 |
| mordred | phschwartz: are you using ansible to do the nova api calls for node creation too? you said you're using it for launch_node | 15:39 |
| fungi | since the puppet master also has the ca key and can revoke/replace any keys it wants | 15:39 |
| mordred | yeah. pretty much getting your hands on the puppetmaster though means you rule the world | 15:40 |
| phschwartz | mordred: I am. I have some things that I would have to clean up, but I can share it after I do that. It uses the rax module to create instances and then uses the same module to do dns updates/additions | 15:40 |
| mordred | fungi: since we're talkinga bout it - if you wanted to review those ansible patches I would not be offended :) | 15:40 |
| mordred | phschwartz: ah. well piddle. I was hoping you were using the nova module - but I'd like to see the code whenever you feel like sharing so I could learn from it | 15:41 |
| *** jcoufal has joined #openstack-infra | 15:41 | |
| *** maxbit has quit IRC | 15:41 | |
| mordred | (we have other people running infra not on rax, so we try to stay away from using _anything_ that's not 100% openstack - rax dns is one thing where we're making an exception at the moment, but we're not 100% thrilled about it | 15:42 |
| phschwartz | I can pretty easily convert it to the nova module, but that doesn't get us the dns handling. That would still need the rax module | 15:42 |
| phschwartz | Let me make some changes to it and I will move it to use the nova module and move the dns to a segregated role that uses the rax module. | 15:43 |
| mordred | yeah. that sounds like a win | 15:43 |
| mordred | I mean, our script right now prints out rax-dns commands | 15:43 |
| fungi | basically the *only* reason we're using the rax dns api client is because there is no official openstack dnsaas _yet_ | 15:43 |
| * mordred is looking forward to rax moving to designate | 15:43 | |
| phschwartz | mordred: you and me both. We have an unrestricted dns api account for our deployments to use, but if we have to recreate a large portion of our control plain, we end up flooding the dns api for our apikey and have to throttle back. | 15:45 |
| mordred | haha. nice | 15:45 |
| phschwartz | But we are talking about a thousand+ of requests in a min | 15:45 |
| *** denis_makogon has joined #openstack-infra | 15:46 | |
| mordred | believe me - we know exactly what you mean WRT flooding cloud apis :) | 15:46 |
| fungi | sdague: gate stability seems to have improved dramatically | 15:46 |
| mordred | fungi: also, I'd love your opinion as a greybeard on https://review.openstack.org/#/c/92677 - the code works, I think the outstanding questio is really "should we" | 15:47 |
| anteaya | fungi: yay | 15:47 |
| fungi | mordred: can we hook that up to an sms gateway? | 15:48 |
| sdague | fungi: yeh, so that indicates that basically all the failures are the fact that after nova-network runs a certain number of computes, it goes silly | 15:48 |
| fungi | sdague: makes sense. i was wondering how that one revert accomplished that (server reuse sounded like something which would have caused *more* problems, not less) | 15:49 |
| fungi | mordred: so if we do this, should we actively try to limit our status updates to 120 characters to prevent truncation/splitting? | 15:50 |
| jogo | fungi: I see pypi.openstack.org is still not automatically updating after a new hacking release http://pypi.openstack.org/openstack/hacking/ latest is 0.9.1 | 15:51 |
| mordred | fungi: nah - I think we should do status updates that are what we want to say to IRC - I'd see it purely as an additional way for us to get the word out - because we _still_ have people who don't notice when we send out updates | 15:51 |
| fungi | jogo: i'll check the log on it. i ironed a bunch of bugs out of the new wheel uploading mechanisms last week | 15:51 |
| *** jlibosva has quit IRC | 15:51 | |
| fungi | but there could still be more | 15:51 |
| *** david-lyle has joined #openstack-infra | 15:51 | |
| *** bauzas has joined #openstack-infra | 15:52 | |
| jogo | fungi: cool, thanks | 15:52 |
| fungi | jogo: did you maybe need to force a refresh? looks there to me | 15:52 |
| jogo | fungi: that is the last blocker to release hacking 0.9.1 | 15:53 |
| jogo | fungi: I see 0.8.1 not 0.9.1 | 15:53 |
| fungi | jogo: uploaded last night (05:50 utc) | 15:53 |
| jogo | at http://pypi.openstack.org/openstack/hacking/ | 15:53 |
| *** e0ne has quit IRC | 15:53 | |
| fungi | jogo: oh, on the mirror | 15:53 |
| fungi | durr | 15:53 |
| fungi | i read that as pypi.python.org | 15:54 |
| fungi | checking the mirror job logs now ;) | 15:54 |
| jogo | oh haha thanks | 15:54 |
| *** maxbit_ has quit IRC | 15:54 | |
| *** e0ne has joined #openstack-infra | 15:54 | |
| *** mrodden has quit IRC | 15:54 | |
| *** maxbit has joined #openstack-infra | 15:54 | |
| fungi | so the *good* news is that wheel+tarball uploads to pypi seem to be reliable now (except for data-only projects with no python modules in them) | 15:55 |
| fungi | (which is a bug in wheels, which i need to get around to filing) | 15:55 |
| jogo | ohh python | 15:55 |
| *** Hal_ has quit IRC | 15:56 | |
| mordred | fungi: can we loop up on the data-only projects issue | 15:56 |
| mordred | I beleive Iw as supposed to look in to work arounds) | 15:56 |
| fungi | jogo: working as designed... | 15:56 |
| fungi | jogo: http://git.openstack.org/cgit/openstack/requirements/tree/global-requirements.txt#n140 | 15:56 |
| fungi | mordred: i build a very trivial workaround (proposed to one of tie in https://review.openstack.org/98456 after testing it works) | 15:57 |
| fungi | er, built | 15:57 |
| fungi | mordred: but it looks like there's more interest in seeing it fixed in the wheel package instead | 15:58 |
| mordred | fungi: gotcha. cool | 15:58 |
| mordred | fungi: if we've got a simple workaround (even if it's ugly) and upstream sees it as a bug | 15:58 |
| jogo | fungi: ohh I thought we had as special clause for hacking, but this makes more sense | 15:58 |
| *** e0ne has quit IRC | 15:58 | |
| fungi | jogo: yes, if newer hacking isn't allowed in global-reqs, we're not going to mirror it | 15:59 |
| jgriffith | mriedem: fungi either of you have an objection to me creating a non-voting job for cinder conf up to date check? | 15:59 |
| mordred | fungi: we _could_ potentially put in a thing to pbr that would detect a lack of package and if so add a blank file | 15:59 |
| *** gokrokve has joined #openstack-infra | 15:59 | |
| fungi | jgriffith: sounds like a fine idea to me | 15:59 |
| mordred | fungi: but leaving well enough alone also seems like a good idea | 15:59 |
| jgriffith | mriedem: fungi it seems there will be a revolt if I remove it entirely | 15:59 |
| *** andreaf has quit IRC | 16:00 | |
| jgriffith | fungi: good enough for me... I'll see what I can work out | 16:00 |
| fungi | mordred: yeah, dstufft agreed it's a bug, i was just too far under water last week to find time to file it | 16:00 |
| mordred | ++ | 16:01 |
| sdague | fungi: more importantly that server reuse patch actually passed through check and gate fine, which means it passed tempest running those tests at least 10 times with no fails | 16:01 |
| mordred | fungi: with https://review.openstack.org/#/c/96636 - it makes me wonder if, if we do go down this path (which rax also being down the path makes me even more excited to do) - whether we shouldn't split out that tree into its own tree that we git sync similar to but different from infra/config | 16:01 |
| mordred | sdague: nice! | 16:02 |
| mordred | sdague: what's the tl;dr on the server reuse patch? I tried to find the convo in the scrollback but failed... | 16:02 |
| fungi | sdague: not only that, it was rechecked several times and passed every one | 16:02 |
| sdague | mordred: well, not nice, because reverting it made things better | 16:02 |
| sdague | fungi: no, I mean the initial patch | 16:02 |
| sdague | that was reverted | 16:02 |
| fungi | sdague: oh! i thought you meant the revert | 16:03 |
| sdague | fungi: nope | 16:03 |
| fungi | weird | 16:03 |
| sdague | I went back to make sure it wasn't one of the ones I asked you to ninja merge | 16:03 |
| sdague | and it wasn't | 16:03 |
| sdague | mordred: the actions tests in tempest reuse a guest (and rebuild it if it looks funky) because they all do odd things to guests and so would be otherwise expensive | 16:04 |
| sdague | I thought that was adding variability, so made it not do that and start a new guest each time | 16:04 |
| sdague | and that all passed | 16:04 |
| *** mbacchi has quit IRC | 16:04 | |
| sdague | then sun spots shifted | 16:04 |
| *** thedodd has quit IRC | 16:04 | |
| *** IvanBerezovskiy has left #openstack-infra | 16:04 | |
| sdague | and nova network stopped being able to create networks in any vaguely reliable way | 16:05 |
| sdague | and reverting that patch seems to have made it better | 16:05 |
| fungi | sdague: though actually, that patchset failed a couple of tempest jobs with unidentified failures in resize tests, and dkranz did a recheck no bug on it | 16:05 |
| *** maxbit_ has joined #openstack-infra | 16:05 | |
| sdague | fungi: they were identified resize failures | 16:05 |
| fungi | aha | 16:05 |
| sdague | just not in the patch stream | 16:05 |
| openstackgerrit | Joe Gordon proposed a change to openstack/requirements: Add hacking 0.9.0 series https://review.openstack.org/98824 | 16:05 |
| jogo | fungi: ^ I think that should do it | 16:05 |
| sdague | we had a resize patch fix running in parallel | 16:05 |
| fungi | got it | 16:06 |
| *** maxbit has quit IRC | 16:06 | |
| *** david-lyle has quit IRC | 16:06 | |
| mordred | sdague: wow. | 16:06 |
| mordred | sdague: this makes me want to call elastic-recheck "sunspots" btw | 16:07 |
| *** mrodden has joined #openstack-infra | 16:07 | |
| *** ramashri has joined #openstack-infra | 16:07 | |
| *** godp13011 has quit IRC | 16:07 | |
| sdague | :) | 16:07 |
| mriedem | fungi: yeah we got the resize/migration failures fixed last week via another revert | 16:07 |
| fungi | oh, awesome | 16:08 |
| *** wenlock has quit IRC | 16:08 | |
| *** ildikov_ has joined #openstack-infra | 16:08 | |
| *** vhoward has joined #openstack-infra | 16:09 | |
| sdague | so... here is the suspicious part | 16:09 |
| openstackgerrit | Philip Marc Schwartz proposed a change to openstack-infra/config: Creation of vinz project in the openstack-infra scheme. https://review.openstack.org/93953 | 16:10 |
| sdague | after the revert of mriedem's patch - http://logstash.openstack.org/#eyJzZWFyY2giOiJidWlsZF9zaG9ydF91dWlkOjEwNWRiNjggQU5EIG1lc3NhZ2U6XCJJbnN0YW5jZSBzcGF3bmVkXCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjYwNDgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjE0MDIzMzAxNjA5Mjl9 | 16:10 |
| sdague | sorry, mriedem's revert of my patch | 16:10 |
| phschwartz | anteaya: fungi: I think I have the issue fixed, lets wait for check to run and see. | 16:10 |
| *** yamahata has joined #openstack-infra | 16:10 | |
| sdague | we spawn 230 guests in a tempest full run | 16:10 |
| *** ildikov has quit IRC | 16:10 | |
| sdague | before, http://logstash.openstack.org/#eyJzZWFyY2giOiJidWlsZF9zaG9ydF91dWlkOjg1MTJmZTIgQU5EIG1lc3NhZ2U6XCJJbnN0YW5jZSBzcGF3bmVkXCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjYwNDgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjE0MDIzMzAwNTU4MzJ9 | 16:11 |
| *** zns has joined #openstack-infra | 16:11 | |
| sdague | 278 | 16:11 |
| sdague | and with a magic number in between | 16:11 |
| fungi | and here i thought 3 was a magic number | 16:11 |
| arunkant | Hi..still waiting for merge of https://review.openstack.org/#/c/95842/. It was code reviewed by keystone core and infra core earlier...still waiting for this merge to happen as this one blocking another keystone change. | 16:11 |
| sdague | :) | 16:11 |
| fungi | though i guess 256 wouldn't have worked as well in the song | 16:11 |
| mordred | sdague: wait - you're telling me that we're tripping ... a 256 problem? | 16:12 |
| mordred | in python code? | 16:12 |
| mordred | what, are we doing bitshifting? | 16:12 |
| fungi | <<24 | 16:12 |
| *** yfried has joined #openstack-infra | 16:13 | |
| *** Samolo_ has quit IRC | 16:13 | |
| *** yfried has quit IRC | 16:13 | |
| mordred | fungi: please tell me you're just kidding | 16:13 |
| *** yfried has joined #openstack-infra | 16:13 | |
| mriedem | 256 == fixed ips? | 16:14 |
| fungi | mordred: i am just kidding | 16:14 |
| mordred | wow. you are not kidding | 16:14 |
| fungi | but yes, we're inheriting from ye olde ipv4 | 16:14 |
| *** jlk has quit IRC | 16:14 | |
| mordred | fungi: go "git grep '<<'" in the nova repo | 16:14 |
| zaro | morning | 16:14 |
| mordred | nova/ipv6/account_identifier.py: static_num = netaddr.IPAddress(0xff << 24) | 16:14 |
| jogo | haha | 16:14 |
| fungi | mordred: now you have me afraid | 16:14 |
| *** jlk has joined #openstack-infra | 16:14 | |
| mordred | nova/ipv6/account_identifier.py: int(hashlib.sha1(project_id).hexdigest()[:8], 16) << 32) | 16:15 |
| mordred | nova/tests/virt/vmwareapi/test_driver_api.py: mem_kib = long(self.type_data['memory_mb']) << 10 | 16:15 |
| fungi | eek! there should already be a primitive for that | 16:15 |
| *** sarob has joined #openstack-infra | 16:15 | |
| mordred | fungi: now, luckily, the ipv6 dir is the only place that we're doing that that's not in extension land | 16:15 |
| fungi | if you're bitshifting with crypto libraries, "you're doing it wrong"[tm] | 16:15 |
| mordred | uhm. yeah | 16:16 |
| *** jgrimm has joined #openstack-infra | 16:16 | |
| sdague | mordred: network_size = 256 | 16:16 |
| fungi | also, seriously, someone did <<10 when they could have just added *1024?!? madness | 16:16 |
| jogo | sdague: so how do folks use more then 256 instances with nova network? | 16:16 |
| *** jlk has quit IRC | 16:16 | |
| *** che-arne has quit IRC | 16:16 | |
| *** che-arne has joined #openstack-infra | 16:17 | |
| mordred | jogo: who needs more than 256 instances? | 16:17 |
| sdague | jogo: I don't know for sure that it's the issue | 16:17 |
| mordred | nova/tests/test_nova_manage.py: network_size=256, | 16:17 |
| *** pdmars has joined #openstack-infra | 16:17 | |
| sdague | however, the fact that our failure rate went way up once we crossed that boundary, and way down once we came back, is suspicious | 16:18 |
| mordred | ++ | 16:18 |
| *** _nadya_ has joined #openstack-infra | 16:18 | |
| sdague | especially given that the changes *should* have made us more reliable | 16:18 |
| sdague | as we were using a clean guest each time | 16:18 |
| *** _nadya_ has quit IRC | 16:18 | |
| jogo | yeah. I wonder if vishy is around this week. He knows more about nova-net then anyone else | 16:18 |
| fungi | sdague: i guess the question, sparked by mriedems comment, is have we hard-coded a /24 somewhere in devstack/tempest which is getting filled | 16:18 |
| sdague | fungi: actually I think we don't set anything, and nova default is 256 | 16:19 |
| fungi | aha | 16:19 |
| *** Longgeek has quit IRC | 16:19 | |
| fungi | and that sounds like something which should be fixed (why would that even have a default? wouldn't almost everyone need to change it?) | 16:19 |
| *** sarob has quit IRC | 16:20 | |
| openstackgerrit | Marton Kiss proposed a change to openstack-infra/config: Add md5 checksum to groups tarball publisher https://review.openstack.org/97238 | 16:20 |
| sdague | honestly, I think it's fine as a default | 16:20 |
| sdague | the real probably is you can't read a n-net log - http://logs.openstack.org/05/98505/4/check/check-tempest-dsvm-full/e6e28c2/logs/screen-n-net.txt.gz | 16:20 |
| fungi | i think 2^64 (one ipv6 network) makes a more sensible forward-looking default ;) | 16:20 |
| *** markmcclain has joined #openstack-infra | 16:21 | |
| sdague | fungi: also realize that all the ips have to be preallocated, so that's super painful | 16:21 |
| *** jaypipes has quit IRC | 16:21 | |
| derekh_ | sdague: about https://review.openstack.org/#/c/98154/ we fully intend to add tempest to our overcloud job in the near future (we're currently fucosed on the reliability of the image builds and simple deployment) | 16:21 |
| fungi | oh, ouch. okay that sounds like a design flaw too | 16:21 |
| *** che-arne has quit IRC | 16:21 | |
| *** sarob has joined #openstack-infra | 16:21 | |
| *** zzelle_ has joined #openstack-infra | 16:21 | |
| vishy | jogo: sup? | 16:22 |
| fungi | i guess the goal is to let neutron solve this problem | 16:22 |
| *** yamahata has quit IRC | 16:22 | |
| jogo | vishy: ^ read backlog, nova-network issues | 16:22 |
| jogo | sdague: ^ | 16:22 |
| fungi | in which case nova-net defaults seem a little less worrisome in the face of ipv6land | 16:22 |
| *** yamahata has joined #openstack-infra | 16:22 | |
| fungi | forward-looking and nova-network probably don't really need to be found in the same sentence | 16:22 |
| vishy | jogo: just skimming but is the issue that you can’t create a network larger than 256 addresses? | 16:23 |
| *** aconrad_ has joined #openstack-infra | 16:23 | |
| ArxCruz | Who ping me? | 16:23 |
| ArxCruz | Can't see in the logs :/ | 16:23 |
| jogo | vishy: with the default network_size=256 yes. | 16:23 |
| jogo | sdague dug into this the most | 16:24 |
| sdague | jogo: no, you misread it | 16:24 |
| vishy | because I noticed a “feature” recently in nova network that you have to specify a larger network_size to create something bigger | 16:24 |
| fungi | ArxCruz: anteaya, about 1.5 hours ago | 16:24 |
| vishy | which makes no sense | 16:24 |
| ArxCruz | fungi: thanks | 16:24 |
| vishy | hold on one sec brb | 16:24 |
| ArxCruz | anteaya: sorry, had to head to dentist did I miss the meeting ? | 16:25 |
| sdague | vishy: we noticed the reliability of tests dropped dramatically after we created > 250 guests during a test run | 16:25 |
| jogo | sdague: woops, I did | 16:25 |
| sdague | not all at at once | 16:25 |
| *** sballe has joined #openstack-infra | 16:25 | |
| sdague | just over the life of the test run | 16:25 |
| sdague | which is really suspicious | 16:25 |
| *** signed8bit is now known as signed8bit_ZZZzz | 16:25 | |
| *** sarob has quit IRC | 16:25 | |
| vishy | ok back | 16:26 |
| *** pdmars has quit IRC | 16:26 | |
| vishy | so is the network larger than 256? | 16:26 |
| sdague | no | 16:26 |
| *** pblaho has quit IRC | 16:26 | |
| vishy | i wouldn’t be surprised if we are running out of fixed ips | 16:26 |
| sdague | vishy: we're deleting the guests | 16:26 |
| vishy | especially if the fixed ip release code is failing somehow | 16:26 |
| sdague | probably never more than 4 at a time are up | 16:26 |
| sdague | right | 16:26 |
| *** praneshp has joined #openstack-infra | 16:26 | |
| sdague | that's my suspicion | 16:26 |
| vishy | that has happened before :) | 16:26 |
| vishy | did this just start recently? | 16:26 |
| vishy | do you have logs from a failed run? | 16:27 |
| *** jlk has joined #openstack-infra | 16:27 | |
| sdague | http://logs.openstack.org/98/98098/2/gate/gate-tempest-dsvm-full/8512fe2/ - failed run | 16:27 |
| sdague | vishy: well we crossed that boundary only very recently | 16:28 |
| *** david-lyle has joined #openstack-infra | 16:28 | |
| *** basha has quit IRC | 16:28 | |
| *** wenlock has joined #openstack-infra | 16:30 | |
| *** melwitt has joined #openstack-infra | 16:30 | |
| vishy | sdague: lease/release looks fine there | 16:31 |
| pleia2 | morning | 16:31 |
| *** medieval1 has quit IRC | 16:32 | |
| *** alkari has joined #openstack-infra | 16:33 | |
| sdague | vishy: yeh, like I said it's mostly suspicious | 16:33 |
| mordred | sdague: yah - if something is broken in ip delete, then you'd really only see that if you made more nodes than the network size over the run life | 16:33 |
| vishy | what was the actual failure? | 16:33 |
| *** briancline has quit IRC | 16:33 | |
| mordred | and, just for the record, as consumers of openstack, we've certainly run in to problem with delete not delting things :) | 16:33 |
| *** briancline has joined #openstack-infra | 16:33 | |
| Shrews | i hate when my things aren't delted :) | 16:34 |
| * Shrews pokes mordred in the eye | 16:35 | |
| *** medieval1 has joined #openstack-infra | 16:35 | |
| sdague | vishy: at some point we couldn't ssh into a guest to verify it | 16:35 |
| *** yamahata has quit IRC | 16:35 | |
| *** yamahata has joined #openstack-infra | 16:36 | |
| sdague | and the likelyhood of that happening seems to be much higher if we allocate more than 256 guests | 16:36 |
| vishy | that is a really tough issue to debug | 16:36 |
| vishy | without having a failed env up | 16:36 |
| *** lcheng has quit IRC | 16:36 | |
| *** jlk has quit IRC | 16:36 | |
| vishy | there could be so many things going wrong | 16:36 |
| *** jlk has joined #openstack-infra | 16:37 | |
| vishy | could be iptables, stale mac cache, hypervisor failure, etc. | 16:37 |
| *** jlk has quit IRC | 16:37 | |
| *** jlk has joined #openstack-infra | 16:37 | |
| sdague | right, this is the reason I was trying to drum up someone to make n-net be a bit more deliberate about logging what's going on :) | 16:38 |
| *** jlk has quit IRC | 16:38 | |
| *** pdmars has joined #openstack-infra | 16:38 | |
| *** jlk has joined #openstack-infra | 16:39 | |
| *** aconrad_ has quit IRC | 16:39 | |
| vishy | sdague: it seems odd that we are using 172.24.4.1 | 16:40 |
| *** jcoufal has quit IRC | 16:40 | |
| vishy | as a floating ip | 16:41 |
| *** rcarrill` has joined #openstack-infra | 16:42 | |
| *** david-lyle has quit IRC | 16:43 | |
| *** trinaths has quit IRC | 16:43 | |
| *** sarob has joined #openstack-infra | 16:44 | |
| *** rcarrillocruz has quit IRC | 16:44 | |
| *** maxbit has joined #openstack-infra | 16:44 | |
| *** jistr has quit IRC | 16:44 | |
| *** basha has joined #openstack-infra | 16:45 | |
| clarkb | o/ | 16:46 |
| mordred | look, it's a clarkb | 16:46 |
| notmyname | good morning -infra | 16:47 |
| notmyname | I saw late last week that there were some long gate queues. has that been cleaned up? looks shorter today. is that because it's fixed or because of the weekend? | 16:47 |
| *** reed has joined #openstack-infra | 16:48 | |
| *** maxbit_ has quit IRC | 16:48 | |
| vishy | sdague: I’m suspecting stale arp cache | 16:48 |
| vishy | where do we connect from? | 16:48 |
| vishy | i note that send_arp_for_ha=False in our config | 16:48 |
| vishy | we are reusing the same floating ip for a bunch of tests | 16:49 |
| *** sarob has quit IRC | 16:50 | |
| *** basha has quit IRC | 16:50 | |
| mordred | notmyname: a little of both - it's still being worked | 16:51 |
| mtreinish | vishy: it's all run on the same host | 16:51 |
| notmyname | mordred: ok, thanks | 16:51 |
| *** yaguang has quit IRC | 16:52 | |
| vishy | mtreinish: hmm that’s odd | 16:52 |
| mordred | clarkb: ooh, neat. the fine folks at stackalytics (thanks ilyashakhat) merged most of my patches over the weekend - except for one which had a bug | 16:52 |
| clarkb | mordred: nice | 16:52 |
| notmyname | mordred: I'm looking at a 28-patches long dependency chain being merged this week. based on the numbers from last friday, that would take about 6 months. so you can understand how I'm worried | 16:52 |
| *** lakshmiS has joined #openstack-infra | 16:52 | |
| clarkb | before I dive into more dib this morning isthere naything I can do to help with the gate triage? | 16:53 |
| mtreinish | vishy: it's run in the gate slaves, so devstack on a single node which also runs the tempest tests | 16:53 |
| vishy | mtreinish: and we don’t get the console log from the vms anwhere do we? | 16:53 |
| mordred | notmyname: yes indeed | 16:53 |
| mtreinish | vishy: on some failures we do, I have a patch out which fixes it for the majority case: https://review.openstack.org/#/c/98446/ | 16:54 |
| vishy | sdague: i think i need the vm console logs (and preferrably syslog) to have a chance to debug this | 16:54 |
| sdague | vishy: syslog for the box is there | 16:55 |
| clarkb | pleia2: that your for the e-r docs | 16:55 |
| *** sarob has joined #openstack-infra | 16:55 | |
| vishy | sdague: i need for the vm | 16:55 |
| pleia2 | clarkb: sure | 16:55 |
| sdague | vm console logs is something we're working on | 16:55 |
| mtreinish | vishy: here is a console log from a ssh timeout failure: http://logs.openstack.org/46/98446/2/check/check-tempest-dsvm-full/5e97e50/console.html.gz#_2014-06-06_18_56_04_177 | 16:55 |
| notmyname | mordred: so that being said, I'm not sure what the possibilities are. one thing I've thought of (but don't know if it works) is to have one merge commit for the chain and only test/gate that one. but our current automatic tooling doesn't support that | 16:55 |
| mordred | notmyname: ah, but it does | 16:56 |
| sdague | so this is another curiosity - eyJzZWFyY2giOiJcImxhenkgbG9hZCBvcGVyYXRpb24gb2YgYXR0cmlidXRlICdmaXhlZF9pcCcgY2Fubm90IHByb2NlZWRcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTQwMjMzMjg4MTYyNH0 | 16:56 |
| clarkb | mordred: you don't have a link to your slave DB changes handy do you? | 16:56 |
| mordred | clarkb: one sec | 16:56 |
| sdague | those are all ERROR at success | 16:56 |
| notmyname | mordred: tell me more :-) | 16:56 |
| clarkb | mordred: I should start there as my next steps are to dib unittest slaves | 16:56 |
| openstackgerrit | Philip Marc Schwartz proposed a change to openstack-infra/config: Creation of vinz project in the openstack-infra scheme. https://review.openstack.org/93953 | 16:56 |
| mordred | clarkb: https://review.openstack.org/#/c/98586/2 | 16:56 |
| clarkb | danke | 16:56 |
| mtreinish | sdague: broken link... | 16:56 |
| mordred | notmyname: if your sequence of things is in a branch | 16:57 |
| phschwartz | god, my hands typing hate me today | 16:57 |
| sdague | mtreinish: search for - "lazy load operation of attribute 'fixed_ip' cannot proceed" | 16:57 |
| mordred | notmyname: then we can make a merge commit to master of that branch | 16:57 |
| mordred | notmyname: and then all of the tooling will DTRT | 16:57 |
| vishy | sdague: i think it couldn’t hurt to turn on send_arp_for_ha, might be nice to see if it helps | 16:58 |
| *** godp1301 has joined #openstack-infra | 16:58 | |
| mordred | notmyname: you re-worked the patch series into a more readable series for review from what was in your feature branch, right? | 16:58 |
| notmyname | mordred: but to get the chain into gerrit, it has to pass all the gate tests anyway, right? (I'm not worried about check tests since those are parallel) | 16:58 |
| sdague | vishy: sure | 16:58 |
| notmyname | mordred: correct. it's not a branch with a merge commit to master now, but it could be. right now it's just a set of patches to master with dependencies | 16:59 |
| mordred | notmyname: we can force-push the tip of your changes to the feature/ec branch which will get the refs into gerrit | 16:59 |
| mordred | notmyname: then make the merge commit to master which would be the thing that would have to pass tests | 16:59 |
| *** wenlock has quit IRC | 16:59 | |
| *** derekh_ has quit IRC | 16:59 | |
| mordred | notmyname: so it would still need to be tested before landing on swift master | 16:59 |
| notmyname | mordred: ah. and clarkb won't come stab me for that? ;-) | 16:59 |
| mordred | notmyname: he may come stab you. he's shifty | 16:59 |
| clarkb | mordred: I wanted to avoid breaking history on that branch | 17:00 |
| notmyname | mordred: right. and in fact, the set of patches are for reviewable chunks. but really all anyone is concerned about is the final result of all of them. | 17:00 |
| clarkb | mordred: the mp branches have taught us our lesson about that | 17:00 |
| *** matjazp has joined #openstack-infra | 17:00 | |
| mordred | clarkb: yah. but in this case, I believe this particular set of patches has gone through review and testing so far, no? | 17:00 |
| mordred | notmyname: right | 17:00 |
| clarkb | mordred: not if you are force pushing over them | 17:01 |
| clarkb | the code in feature/ec currently has gone through that process | 17:01 |
| notmyname | clarkb: mordred: yes, somewhat. the reorged patches that are currently proposed are what are getting full review from the community | 17:02 |
| mordred | clarkb: aiui, this is the same code, just broken up into different chunks for ease of review | 17:02 |
| mordred | I mean, we could ALSO just propse a merge commit of the feature/ec branch in its current state | 17:03 |
| notmyname | mordred: correct | 17:03 |
| notmyname | mordred: no, we don't want that | 17:03 |
| mordred | and use the 17 patch re-working as the place for review | 17:03 |
| mordred | k | 17:03 |
| * mordred just thinking out loud right now | 17:03 | |
| clarkb | mordred: my issue is less with review and testing and more with moving a branch pointer to historically different code breaks in funny ways | 17:04 |
| clarkb | and isn't really nice to downstream cloners | 17:04 |
| mordred | clarkb: we could make a new branch | 17:04 |
| reed | is access to our gerrit via API documented somewhere? | 17:04 |
| clarkb | mordred: that would be fine by me | 17:05 |
| reed | checking ci.openstack.org, can't find it | 17:05 |
| mordred | clarkb: "feature/ec-review" - and push the tip of the review series to that once the review series has been fully +2'd | 17:05 |
| clarkb | reed: yes, if you click along the top bar of gerrit there is a Documentation tab | 17:05 |
| mordred | notmyname: ^^ | 17:05 |
| *** aconrad_ has joined #openstack-infra | 17:05 | |
| reed | ah, interesting, clarkb thanks... only available to authenticated... now I see it :) | 17:06 |
| clarkb | reed: and from there you can get to https://review.openstack.org/Documentation/rest-api.html | 17:06 |
| notmyname | mordred: clarkb: well on other logistics issue. the current feature/ec branch doesn't have _any_ erasure code stuff on it. and in fact, we'd like to do the erasure code work the same way, and we'd prefer the feature/ec name there since it's a good name for what we'll actually be doing. that's a separate conversation for later, but FYI | 17:06 |
| clarkb | reed: it should be available unauthenticated too | 17:06 |
| mordred | notmyname: kk | 17:06 |
| mordred | notmyname: I'm sure we can figure out something :) | 17:06 |
| * mordred needs to AFK for a bit | 17:06 | |
| reed | the url seems to be, the link 'documentation' appeared only after I logged in | 17:06 |
| mordred | reed: awesome | 17:06 |
| notmyname | mordred: clarkb: our current goal is to approve on Wednesday | 17:06 |
| clarkb | reed: I am not logged in and it is there for me | 17:06 |
| reed | uhm | 17:07 |
| mordred | notmyname: ok. just come poke us in here and we can work with you on the actual logistics | 17:07 |
| notmyname | mordred: ok, so to make sure I understand what you were proposing.... | 17:07 |
| reed | weird | 17:07 |
| * reed needs more coffee | 17:07 | |
| *** marun has joined #openstack-infra | 17:07 | |
| *** pdmars has quit IRC | 17:07 | |
| notmyname | mordred: continue to review the current patch chain. then create feature/sp-review (note s/ec/sp). then propose merge commit and approve that | 17:08 |
| *** lcheng has joined #openstack-infra | 17:08 | |
| *** mmaglana has joined #openstack-infra | 17:09 | |
| *** amcrn has joined #openstack-infra | 17:09 | |
| *** harlowja_away is now known as harlowja | 17:09 | |
| notmyname | mordred: what if we just reproposed the patch chain to a feature/sp-review now? and worked from there? either way, how do we get the patches (not the merge commit) into gerrit without having to wait 6 months for the gate? | 17:09 |
| *** annegentle_ has joined #openstack-infra | 17:11 | |
| *** marun has quit IRC | 17:11 | |
| *** thedodd has joined #openstack-infra | 17:12 | |
| vishy | sdague: another useful thing that would be nice to have on failure would be the output of sudo iptables -nvL and sudo iptables -t nat -nvL and sudo ip addr and sudo brctl show | 17:12 |
| vishy | in other words a dump of all networking info from the box | 17:12 |
| sdague | vishy: yeh, we do that some times. Definitely trying to think about a centralized location for postmortem dump so we can do that consistently | 17:13 |
| sdague | it's been mostly ad-hoc applied in really troublesome tests at this point | 17:13 |
| anteaya | mordred: can you take a minute and respond to this please? http://lists.openstack.org/pipermail/openstack-dev/2014-June/037025.html | 17:14 |
| anteaya | I'm really not a fan of different groups of third party ci folks doing different things for each project | 17:15 |
| anteaya | I don't seem to be yelling loud enough though | 17:15 |
| anteaya | or anyone else who can stem this line of thought | 17:15 |
| *** tkelsey has quit IRC | 17:15 | |
| *** crc32 has joined #openstack-infra | 17:15 | |
| anteaya | fungi: cool | 17:16 |
| anteaya | ArxCruz: no not yet but I do need to chat with you | 17:16 |
| ArxCruz | anteaya: O.O what I did wrong ? | 17:17 |
| anteaya | nothing | 17:17 |
| ArxCruz | oh, then okay :) | 17:17 |
| sdague | fungi: do we have a dns bug open with rax? | 17:17 |
| sdague | http://logs.openstack.org/98/98098/2/gate/gate-tempest-dsvm-postgres-full/bab1954/ - failure is rax can't resolve mirror.rackspace.com | 17:17 |
| clarkb | mordred: I don't think the change in https://review.openstack.org/#/c/88479/16..18/modules/iptables/manifests/init.pp will work as expected. facter gives us a string not a boolean and puppet conditionals are bong hits. so in_chroot = 'false' would match that same condition I think | 17:17 |
| anteaya | I need you as a guinea pig for a thing we are doing to stream line info for third party ci | 17:17 |
| clarkb | mordred: I can test this though | 17:17 |
| *** _nadya_ has joined #openstack-infra | 17:18 | |
| sdague | clarkb: is there any way to add more capacity to the ES cluster? We keep going into runaway state just when we need that data the most | 17:20 |
| clarkb | sdague: what do you mean by capacity? | 17:20 |
| clarkb | we are already at 6 X 60GB 8Vpcu nodes | 17:20 |
| sdague | processing capacity | 17:20 |
| clarkb | we can add more nodes | 17:20 |
| clarkb | but its not cheap | 17:20 |
| *** wenlock has joined #openstack-infra | 17:21 | |
| clarkb | also what runaway state? | 17:21 |
| clarkb | the logstash job queue? | 17:21 |
| sdague | yeh | 17:21 |
| clarkb | oh so thats actually logstash and we can add more of those nodes as well | 17:21 |
| *** dkliban is now known as dkliban_bbiab | 17:21 | |
| sdague | right now that's what's biting us | 17:21 |
| clarkb | we shouldn't need ES nodes actually I think they are hodling up | 17:21 |
| *** thedodd has quit IRC | 17:21 | |
| sdague | sure | 17:21 |
| clarkb | we just can't deal with the volume of logs produces | 17:21 |
| * clarkb looks at tempest logs :P | 17:22 | |
| *** chianingwang has joined #openstack-infra | 17:22 | |
| sdague | if you want to dump the tempest logs again, I'm fine with it. Though they are trumped by a factor of 3 by ceilometer in syslog | 17:22 |
| clarkb | sdague: right but unfortunately those are the only swift proxy logs we get | 17:22 |
| sdague | yeh, but the swift proxy logs are never helpful anyway | 17:23 |
| notmyname | ?? | 17:23 |
| clarkb | oh in that case should we start with approving your os loganalyze change? | 17:23 |
| clarkb | sdague: you mentioned it was completely untested so I didn't want to dive in with it | 17:23 |
| sdague | clarkb: let me sort out lunch, then I'll test it | 17:23 |
| clarkb | sdague: awesome | 17:24 |
| clarkb | fungi: mordred: we have 16 x 4GB 4vcpu logstash worker nodes | 17:24 |
| clarkb | fungi: mordred what do you think of adding 4 more? | 17:24 |
| sdague | notmyname: the most interesting thing we've every gotten out of swift proxy logs is the 503 error. But they don't provide any operational information about what fails, so they are fundamentally not very useful for debugging things | 17:25 |
| clarkb | the other option is to scale up. but the logstash stuff scales horizontally well so I am partial to that method | 17:25 |
| openstackgerrit | A change was merged to openstack-infra/config: Add basic elastic-recheck docs page https://review.openstack.org/98481 | 17:25 |
| vishy | sdague: looking through the logs i think i may see the issue | 17:27 |
| vishy | it looks like floating ips are associated immediately after the server is created | 17:27 |
| vishy | that may not be safe | 17:27 |
| *** nati_ueno has joined #openstack-infra | 17:27 | |
| vishy | sdague: floating ips should not be associated until after the server is running | 17:28 |
| notmyname | mordred: how would we get the patches onto a feature branch? would that be some -infra operation to push them there once the patch chain is proposed? | 17:28 |
| mriedem | sdague: clarkb: fwiw, i don't think we're using tags:"tempest.txt" in any e-r queries right now - had one last night but it wasn't catching grenade fails on the n-net ssh timeout bug | 17:28 |
| *** james_li has quit IRC | 17:28 | |
| vishy | sdague: ah nm i misread | 17:28 |
| clarkb | notmyname: it is probably easier to create the branch first and push them directly to that branch | 17:29 |
| *** markmcclain has quit IRC | 17:30 | |
| notmyname | clarkb: there is still a little flux in the patches. so getting the chain approved would be less burdensome for getting the patches on a feature branch, right? | 17:30 |
| clarkb | notmyname: I don't think so. proposing a chain to a branch is as easy as git review branchanme | 17:31 |
| clarkb | and as long as you don't approve them until they are actually ready you can reorder and refactor all you want | 17:31 |
| notmyname | clarkb: right. I just don't want to need to have all 27 patches go through the gate check. I just want the last merge commit. | 17:32 |
| clarkb | oh I don't think there is any way around that | 17:32 |
| *** markmcclain has joined #openstack-infra | 17:33 | |
| notmyname | clarkb: that's the issue though. that's the fundamental problem I started talking with mordred about | 17:33 |
| clarkb | but it is also a fundamental git bisectability thing | 17:33 |
| vishy | sdague: definitely does look like it is floating ip related by the log that mriedem sent | 17:33 |
| clarkb | notmyname: probably better to discuss with mordred then | 17:33 |
| notmyname | clarkb: also, I'm ok with keeping the current feature/ec branch for historical reasons, but if so I'd prefer to have it renamed | 17:34 |
| *** slamont has joined #openstack-infra | 17:34 | |
| *** pdmars has joined #openstack-infra | 17:34 | |
| vishy | there is a small possibility there is not enough entropy for the vm to complete the ssh handshake in time | 17:35 |
| vishy | but that seems less likely | 17:35 |
| clarkb | we do install that entropy package on the slaves but the qemu VMs may not have it | 17:35 |
| fungi | clarkb: more logstash workders seems fine if we expect the load to continue | 17:35 |
| mtreinish | vishy: yeah we increased the ssh timeout by about 100sec to test and it still failed at about the same rate so it's probably not that | 17:36 |
| *** pdmars_ has joined #openstack-infra | 17:36 | |
| *** sweston has joined #openstack-infra | 17:36 | |
| *** adrian_otto has joined #openstack-infra | 17:36 | |
| * fungi is kinda afk a bunch this afternoon. landlord for the old place decided to start showing it to prospective tenants while i'm still living here, which kinda gets in the way of working | 17:36 | |
| *** pdmars has quit IRC | 17:36 | |
| *** russell_h has quit IRC | 17:36 | |
| mriedem | fungi: just work from the closet | 17:36 |
| *** amcrn has quit IRC | 17:36 | |
| mriedem | fungi: when they open it to look around, surprise! | 17:36 |
| fungi | mriedem: heh, that would be awesome | 17:37 |
| fungi | but instead of being awesome, i'm going to try and work from the car via netbook over mobile network | 17:37 |
| *** _nadya_ has quit IRC | 17:38 | |
| *** slamont has quit IRC | 17:38 | |
| *** Ryan_Lane has joined #openstack-infra | 17:38 | |
| *** russell_h has joined #openstack-infra | 17:39 | |
| *** chianingwang has quit IRC | 17:39 | |
| *** amcrn has joined #openstack-infra | 17:40 | |
| *** rcarrill` has quit IRC | 17:40 | |
| clarkb | fungi: I really like working from parks on sunny days | 17:40 |
| *** slamont has joined #openstack-infra | 17:40 | |
| *** adrian_otto has quit IRC | 17:40 | |
| clarkb | highly recommended | 17:41 |
| pleia2 | thoughts on modernizing this slide? http://docs.openstack.org/infra/publications/overview/#(31) (templates, some current jobs) or do we want to keep it old style simple? | 17:41 |
| *** rcarrillocruz has joined #openstack-infra | 17:41 | |
| *** russell_h has quit IRC | 17:41 | |
| *** russell_h has joined #openstack-infra | 17:41 | |
| *** dolphm has quit IRC | 17:41 | |
| pleia2 | I'm giving the overview talk as a keynote next week in Croatia | 17:41 |
| *** chianingwang has joined #openstack-infra | 17:41 | |
| clarkb | pleia2: the potential problem with templates is they don't convey much info to someone looking at it for the first time | 17:42 |
| pleia2 | clarkb: yeah | 17:42 |
| clarkb | pleia2: I would stick with that slide as is because it is a bit more meaningful | 17:42 |
| clarkb | if more verbose | 17:42 |
| *** dolphm has joined #openstack-infra | 17:44 | |
| *** otherwiseguy has quit IRC | 17:46 | |
| lifeless | mordred: I could use a hand with this weird pbr on python26 issue at the bottom of my stack | 17:47 |
| *** dolphm has quit IRC | 17:48 | |
| boris-42 | clarkb h ithere | 17:48 |
| *** wenlock has quit IRC | 17:49 | |
| *** dolphm has joined #openstack-infra | 17:49 | |
| boris-42 | for some reason I am not core in osprofiler | 17:50 |
| *** marcoemorais has joined #openstack-infra | 17:50 | |
| *** marcoemorais has quit IRC | 17:50 | |
| boris-42 | should I do something else except https://review.openstack.org/#/c/97778/4/modules/openstack_project/files/gerrit/acls/stackforge/osprofiler.config | 17:50 |
| *** marcoemorais has joined #openstack-infra | 17:51 | |
| boris-42 | lifeless hi, btw are you going to move subunit to stackforge? | 17:51 |
| lifeless | boris-42: no, github | 17:53 |
| jogo | sdague mriedem: wow its nice to see things getting back to not horrible (http://jogo.github.io/gate/) | 17:55 |
| boris-42 | lifeless why? | 17:55 |
| mriedem | not horrible is not horrible | 17:57 |
| mriedem | agreed | 17:57 |
| lifeless | boris-42: because stackforge is part of openstack, and subunit is generic; because my subunit collaborators don't like gerrit; because latency on getting stackforge things fixed is behind {all official things} and for small teams latency is a killer. | 17:57 |
| *** ilyashakhat_ has joined #openstack-infra | 17:58 | |
| mtreinish | jogo: there was a period of 100% failure on partial n-cpu grenade? | 17:59 |
| jogo | sdague: perhaps its time to explicitly send an email out saying its ok to start approving code? | 17:59 |
| jogo | mtreinish: it was grenades in general | 17:59 |
| jogo | but yeah bad | 17:59 |
| *** mbacchi has joined #openstack-infra | 17:59 | |
| *** e0ne has joined #openstack-infra | 18:01 | |
| *** mkoderer has quit IRC | 18:02 | |
| *** pdmars_ has quit IRC | 18:03 | |
| *** doug-fish has joined #openstack-infra | 18:04 | |
| openstackgerrit | Philip Marc Schwartz proposed a change to openstack-infra/config: Creation of vinz project in the openstack-infra scheme. https://review.openstack.org/93953 | 18:04 |
| *** gondoi is now known as zz_gondoi | 18:05 | |
| SergeyLukjanov | sdague, oh, I've been on driving lessons | 18:06 |
| boris-42 | lifeless ok | 18:06 |
| SergeyLukjanov | sdague, now, ready for ninja merges | 18:06 |
| SergeyLukjanov | morning folks :) | 18:06 |
| *** e0ne has quit IRC | 18:07 | |
| * clarkb installs squid to make dib go faster | 18:07 | |
| *** pdmars has joined #openstack-infra | 18:08 | |
| *** sarob has quit IRC | 18:09 | |
| sdague | vishy: cool, thanks for looking. I wonder if the larger number of guest creates means we were effectively leaking a floating ip at some point, but the system didn't realize it. | 18:14 |
| sdague | jogo: honestly, we're at complete capacity right now anyway. And I'd like to figure out if things are holding. | 18:15 |
| *** melwitt has quit IRC | 18:15 | |
| *** jaypipes has joined #openstack-infra | 18:15 | |
| *** melwitt has joined #openstack-infra | 18:16 | |
| jogo | holding? | 18:16 |
| *** melwitt has quit IRC | 18:16 | |
| *** mrodden has quit IRC | 18:17 | |
| *** melwitt has joined #openstack-infra | 18:17 | |
| jogo | sdague: good point, about the at capacity part, not sure what you mean by holding | 18:17 |
| sdague | failure rates | 18:18 |
| *** mrodden has joined #openstack-infra | 18:21 | |
| *** penguinRaider has joined #openstack-infra | 18:21 | |
| jogo | sdague: ahh yeah, I agree | 18:22 |
| *** signed8bit_ZZZzz is now known as signed8bit | 18:22 | |
| *** atiwari has joined #openstack-infra | 18:24 | |
| *** dims has quit IRC | 18:25 | |
| *** dims has joined #openstack-infra | 18:25 | |
| *** pelix has quit IRC | 18:26 | |
| *** e0ne has joined #openstack-infra | 18:26 | |
| *** otherwiseguy has joined #openstack-infra | 18:27 | |
| *** ildikov has joined #openstack-infra | 18:27 | |
| *** ildikov|2 has joined #openstack-infra | 18:29 | |
| *** ildikov has quit IRC | 18:29 | |
| *** zz_gondoi is now known as gondoi | 18:29 | |
| *** markmcclain has quit IRC | 18:30 | |
| *** ildikov_ has quit IRC | 18:30 | |
| *** markmcclain has joined #openstack-infra | 18:30 | |
| *** markmcclain has quit IRC | 18:31 | |
| *** markmcclain has joined #openstack-infra | 18:31 | |
| *** nati_ueno has quit IRC | 18:31 | |
| *** andreykurilin_ has quit IRC | 18:32 | |
| *** adrian_otto has joined #openstack-infra | 18:33 | |
| fungi | clarkb: sunny days here at the moment == ~33c and no breeze | 18:34 |
| *** nati_uen_ has joined #openstack-infra | 18:34 | |
| fungi | so as tempting as the park idea is, i'll save it for cooler conditions | 18:35 |
| clarkb | fungi: wow. I should really keep quiet about how nice our weather here can be >_> | 18:35 |
| clarkb | though we put up with drizzle for half the year to make up for it | 18:35 |
| fungi | needless to say, part of the reason i'm moving to the coast next week ;) | 18:36 |
| *** james_li has joined #openstack-infra | 18:36 | |
| *** marcoemorais has quit IRC | 18:36 | |
| *** marcoemorais has joined #openstack-infra | 18:37 | |
| *** adrian_otto has quit IRC | 18:37 | |
| *** thedodd has joined #openstack-infra | 18:37 | |
| *** marcoemorais has quit IRC | 18:37 | |
| *** lcostantino has quit IRC | 18:37 | |
| *** marcoemorais has joined #openstack-infra | 18:37 | |
| *** e0ne has quit IRC | 18:39 | |
| *** e0ne has joined #openstack-infra | 18:40 | |
| *** wenlock has joined #openstack-infra | 18:41 | |
| zaro | mgagne: i don't see a use case for perma link gerrit comments | 18:41 |
| mgagne | zaro: referring to them in email or blog post rants? :D | 18:42 |
| *** dkranz has quit IRC | 18:42 | |
| mgagne | zaro: [citation needed] | 18:42 |
| mgagne | zaro: that's my use case =) | 18:42 |
| clarkb | yeah would be good | 18:42 |
| fungi | mgagne: especially rants | 18:42 |
| clarkb | otherwise you have to say commetn from foo on $date | 18:42 |
| mgagne | clarkb: I looked for an HTML element id but couldn't find any =( | 18:43 |
| *** _nadya_ has joined #openstack-infra | 18:43 | |
| zaro | mgagne: did you try on gerrit-review.googlesource.com | 18:45 |
| *** wenlock has quit IRC | 18:45 | |
| *** afazekas has joined #openstack-infra | 18:45 | |
| mgagne | zaro: haven't found much :-/ | 18:47 |
| *** wenlock has joined #openstack-infra | 18:51 | |
| sdague | fungi / clarkb: what do you guys think of this - https://review.openstack.org/#/c/98815/ ? | 18:52 |
| *** matjazp has quit IRC | 18:52 | |
| *** matjazp has joined #openstack-infra | 18:53 | |
| fungi | sdague: i agree that we shouldn't set that in devstack-gate and let devstack decide what's sane | 18:53 |
| *** matjazp_ has joined #openstack-infra | 18:53 | |
| sdague | fungi: in this case it will go all the way back to the ceilometer defaults, which I think is appriopriate | 18:53 |
| sdague | it will also mean a reduction of ceilometer polling by a factor of 40 | 18:54 |
| *** weshay has quit IRC | 18:54 | |
| fungi | sdague: git blame tells me https://review.openstack.org/78537 originally put it there (just a few months ago) | 18:54 |
| *** ildikov|2 is now known as ildikov | 18:54 | |
| jogo | sdague: unrelated but fun ceilometer patch https://review.openstack.org/#/c/98301/ | 18:55 |
| lifeless | clarkb: if you have a second, I am confused about the error on https://review.openstack.org/#/c/93253/ | 18:55 |
| fungi | sdague: to avoid timeouts on related tempest tests waiting for the interval to elapse | 18:55 |
| *** _nadya_ has quit IRC | 18:55 | |
| sdague | fungi: honestly, if the tests depend on a polling cycle, they are broken | 18:55 |
| clarkb | sdague: I am fine with it | 18:55 |
| clarkb | oh is that why | 18:56 |
| *** matjazp has quit IRC | 18:57 | |
| *** dkranz has joined #openstack-infra | 18:58 | |
| *** lcheng_ has joined #openstack-infra | 18:59 | |
| fungi | was it for this abandoned test from around that timeframe, i wonder? https://review.openstack.org/#/c/80965/3/tempest/api/telemetry/test_telemetry_pollster_api.py | 19:00 |
| *** markmcclain has quit IRC | 19:00 | |
| *** lcheng has quit IRC | 19:01 | |
| *** nati_ueno has joined #openstack-infra | 19:02 | |
| fungi | aha, nope. it's to support https://review.openstack.org/64136 and its children (not yet approved). see comments around march 6th | 19:05 |
| fungi | sdague: you probably want to -1 those on these grounds | 19:05 |
| *** dkranz has quit IRC | 19:06 | |
| fungi | to at least give the ceilo devs a heads up | 19:06 |
| *** nati_uen_ has quit IRC | 19:06 | |
| *** sweston has quit IRC | 19:07 | |
| *** markmcclain has joined #openstack-infra | 19:07 | |
| *** praneshp has quit IRC | 19:08 | |
| sdague | sure | 19:08 |
| *** matjazp_ has quit IRC | 19:09 | |
| *** freyes has joined #openstack-infra | 19:09 | |
| *** matjazp has joined #openstack-infra | 19:09 | |
| *** weshay has joined #openstack-infra | 19:09 | |
| fungi | sdague: and 98815 probably needs to be proposed as an actual revert of 78537 for better bookkeeping | 19:09 |
| *** annegentle_ has quit IRC | 19:10 | |
| *** rcarrill` has joined #openstack-infra | 19:10 | |
| clarkb | fungi: ++ and thanks for the legwork | 19:10 |
| *** melwitt has quit IRC | 19:10 | |
| zzelle_ | hi everyone, | 19:10 |
| *** melwitt has joined #openstack-infra | 19:11 | |
| zzelle_ | it seems you healed the gate ? | 19:11 |
| fungi | zzelle_: yes, through great feats of collaboration, many recent bugs have been squashed and things are now flowing a bit more smoothly again | 19:12 |
| *** mbacchi has quit IRC | 19:12 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/os-loganalyze: construct syslog debug level https://review.openstack.org/97863 | 19:12 |
| *** rcarrillocruz has quit IRC | 19:12 | |
| fungi | though it's still a little twitchu | 19:12 |
| *** vladan has quit IRC | 19:12 | |
| fungi | twitchy | 19:12 |
| Kiall | sdague: Heya - Designate exercises running correctly :) http://logs.openstack.org/88/98788/6/check/gate-designate-devstack-dsvm/98b5704/console.html | 19:12 |
| zzelle_ | fungi, good job guys :) | 19:12 |
| fungi | Kiall: congrats! | 19:12 |
| sdague | Kiall: cool. is there a topic to get through those reviews? | 19:13 |
| *** ilyashakhat_ has quit IRC | 19:13 | |
| fungi | i think now we're just being crushed under the (relatively light) weight of all the changes nobody approved while things were bad | 19:13 |
| sdague | clarkb: https://review.openstack.org/97863 is the os-loganalyze filter with tests | 19:13 |
| *** matjazp has quit IRC | 19:14 | |
| clarkb | sdague: reviewing | 19:14 |
| Kiall | sdague: yes, but there all on the Designate side - The designate core folks are working through reviews of them now :) https://review.openstack.org/#/q/status:open+project:stackforge/designate+branch:master+topic:devstack,n,z | 19:14 |
| *** ihrachyshka has joined #openstack-infra | 19:15 | |
| *** nati_ueno has quit IRC | 19:15 | |
| *** denis_makogon has quit IRC | 19:15 | |
| *** dprince has quit IRC | 19:15 | |
| *** vladan has joined #openstack-infra | 19:17 | |
| *** markmcclain1 has joined #openstack-infra | 19:19 | |
| *** dkranz has joined #openstack-infra | 19:19 | |
| *** palar_ has joined #openstack-infra | 19:19 | |
| *** markmcclain1 has quit IRC | 19:20 | |
| *** markmcclain1 has joined #openstack-infra | 19:20 | |
| sdague | clarkb: oh, I think I forgot the sample file | 19:20 |
| clarkb | sdague: ya was just looking at tests thinking there is no file here :) | 19:20 |
| sdague | I'm working up a time link patch right now | 19:20 |
| sdague | so as soon as I do that one, I'll rebase and get the test file in | 19:20 |
| clarkb | ok | 19:21 |
| lifeless | clarkb: any thoughts? | 19:21 |
| mriedem | i'm probably late to the party, but is logstash kaput right now? | 19:21 |
| clarkb | lifeless: sorry haven't looked at it yet | 19:22 |
| *** annegentle_ has joined #openstack-infra | 19:23 | |
| *** markmcclain has quit IRC | 19:23 | |
| *** wenlock has quit IRC | 19:23 | |
| *** dkliban_bbiab is now known as dkliban | 19:23 | |
| *** palar has quit IRC | 19:23 | |
| clarkb | lifeless: that change edits requirements and adds things that are not compatble with global requirements | 19:23 |
| lifeless | clarkb: but its teh global requirements update change, no ? | 19:24 |
| clarkb | lifeless: http://logs.openstack.org/53/93253/7/check/gate-os-cloud-config-requirements/427e7e7/console.html#_2014-06-09_04_05_51_594 | 19:24 |
| *** wenlock has joined #openstack-infra | 19:24 | |
| clarkb | lifeless: yes | 19:24 |
| clarkb | lifeless: it is a bug where the requirements themselves are not gated in that way | 19:24 |
| lifeless | clarkb: so the global change is making changes that are not allowed per global. | 19:24 |
| clarkb | lifeless: correct | 19:24 |
| *** pdmars has quit IRC | 19:24 | |
| lifeless | clarkb: so what should we do ? | 19:24 |
| *** palar_ has quit IRC | 19:25 | |
| clarkb | there is an open bug somewhere and I thought yolanda was working on it. But basically fix global requirements | 19:25 |
| clarkb | the sphinx update is an actual bug. pretty sure we don't want to do updates that way | 19:25 |
| *** mbacchi has joined #openstack-infra | 19:25 | |
| clarkb | because half your projects install one version. the other half install the other and they are at the mercy of pip dep resolution | 19:26 |
| fungi | mriedem: logstash is backlogstash at the moment | 19:26 |
| mriedem | ok | 19:26 |
| fungi | mriedem: see the "logstash job queue" graph at the bottom of http://status.openstack.org/zuul/ | 19:27 |
| *** e0ne has quit IRC | 19:27 | |
| *** wenlock has quit IRC | 19:28 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/os-loganalyze: construct syslog debug level https://review.openstack.org/97863 | 19:29 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/os-loganalyze: support date linking in syslog https://review.openstack.org/98867 | 19:29 |
| sdague | clarkb: that should have it | 19:29 |
| pleia2 | looks like the chef folks would appreciate some eyes on this one: https://review.openstack.org/#/c/98084/ | 19:29 |
| *** sarob has joined #openstack-infra | 19:29 | |
| *** zns has quit IRC | 19:29 | |
| *** matjazp has joined #openstack-infra | 19:30 | |
| *** MarkAtwood has joined #openstack-infra | 19:30 | |
| *** zns has joined #openstack-infra | 19:31 | |
| *** praneshp has joined #openstack-infra | 19:31 | |
| *** ramashri has quit IRC | 19:34 | |
| *** e0ne has joined #openstack-infra | 19:35 | |
| lifeless | clarkb: oh, what should the sphinx version look like ? | 19:36 |
| clarkb | lifeless: we should increase bounds in overlapping manners | 19:36 |
| clarkb | lifeless: global requirements didn't do this | 19:36 |
| sdague | clarkb: I actually think the issue is global requirements did do it | 19:37 |
| clarkb | lifeless: iirc there was a bug in 1.2.X. so we probably need >=1.something,!1.2.X,<1.3.X | 19:37 |
| clarkb | sdague: they weren't overlapping | 19:37 |
| sdague | I believe there was a misunderstanding with the implementation of the overlapping requirements enforcement | 19:37 |
| clarkb | sdague: there definitely is | 19:38 |
| sdague | so it doesn't run on gr | 19:38 |
| clarkb | sdague: but global reqs is at fault for this particular issue | 19:38 |
| clarkb | it is a real bug | 19:38 |
| clarkb | sdague: correct | 19:38 |
| sdague | ok, but the non overlapping issue was mostly to handle our mirrors | 19:38 |
| sdague | so that's just done and in the past now | 19:38 |
| clarkb | sdague: no that isn't why | 19:39 |
| sdague | well that's why I wanted to put it in :) | 19:39 |
| clarkb | sdague: overlapping is required to make devstack work | 19:39 |
| clarkb | otherwise what happens is you install everything and end up with pip installing version X but half your projects need version Y and refuse to work | 19:39 |
| sdague | oh, so this is a different issue | 19:39 |
| clarkb | without overlapping requirements there is no way to move the entire project to the new stuff | 19:40 |
| *** sarob has quit IRC | 19:40 | |
| *** thedodd has quit IRC | 19:40 | |
| *** sarob has joined #openstack-infra | 19:40 | |
| *** markmcclain1 has quit IRC | 19:42 | |
| sdague | ok, fair | 19:42 |
| *** lcheng_ has quit IRC | 19:44 | |
| *** sarob has quit IRC | 19:44 | |
| *** mmaglana_ has joined #openstack-infra | 19:50 | |
| *** lcheng has joined #openstack-infra | 19:50 | |
| phschwartz | anteaya: fungi: if you could be so kind as looking at https://review.openstack.org/#/c/93953/ now. It should be ready for review. | 19:51 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard: Removed PostGres from code and documentation https://review.openstack.org/98870 | 19:52 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/devstack-gate: Revert "Set CEILOMETER_PIPELINE_INTERVAL to 15" https://review.openstack.org/98871 | 19:52 |
| *** mmaglana has quit IRC | 19:52 | |
| *** marun has joined #openstack-infra | 19:55 | |
| clarkb | sdague: well I was going to wait for the os log analyze change to go through jenkins before approving, but it has to pass those tests anyways +2 +A from me | 19:55 |
| sdague | clarkb: sounds good | 19:55 |
| *** mmaglana has joined #openstack-infra | 19:55 | |
| anteaya | phschwartz: *nod, working on a few other things atm, I'm not ignoring you though | 19:55 |
| sdague | yeh, it looks like it's about an hour to allocate nodes in check right now | 19:55 |
| sdague | clarkb: I also think that's part of the logstash backup, our throughput on jobs is much higher given the greater number of jobs per patch | 19:56 |
| sdague | and the check queue is up to 167 | 19:57 |
| *** mmaglana_ has quit IRC | 19:58 | |
| *** ramashri has joined #openstack-infra | 20:02 | |
| *** julim has joined #openstack-infra | 20:03 | |
| *** ramashri_ has joined #openstack-infra | 20:03 | |
| *** ramashri_ has quit IRC | 20:03 | |
| *** ramashri has quit IRC | 20:03 | |
| bknudson | lifeless: 1326504 is the bug for requirements overlap checking | 20:03 |
| *** ramashri has joined #openstack-infra | 20:04 | |
| devananda | sdague: given J1 coming in a few days and that ironic was totaly blocked for the last week, I'd like to try landing a bunch of ironic patches in the next 72 hrs | 20:04 |
| devananda | sdague: does that seem feasible // reasonable? | 20:04 |
| *** amcrn has quit IRC | 20:05 | |
| *** morganfainberg is now known as morganfainberg_Z | 20:06 | |
| lifeless | clarkb: bknudson: ok so I'm still confused; do I a) do nothing and it will fix itself eventually; b) propose some change to global-requirements to unblock stuff; c) run around in a panic; d) ??? | 20:06 |
| *** radez is now known as radez_g0n3 | 20:06 | |
| fungi | lifeless: i think the proposed revert of the original (incorrect) job needs to get some visibility so reqs proposals work again | 20:07 |
| Kiall | fungi / lifeless .. you convince people to +2/+A this https://review.openstack.org/#/c/97893/ ;) | 20:07 |
| lifeless | give me commit access :P | 20:08 |
| *** boris-42 has quit IRC | 20:08 | |
| lifeless | ok so that needs clarkb or jeblair or mordred or SergeyLukjanov or jhesketh :) https://review.openstack.org/#/c/97893/ <- requirements jobs are near-universally failing right now. | 20:08 |
| fungi | yup. anybody with a requirement or test req on sphinx can't update their reqs | 20:12 |
| *** chianingwang_ has joined #openstack-infra | 20:12 | |
| *** boris-42 has joined #openstack-infra | 20:13 | |
| *** matjazp has quit IRC | 20:15 | |
| *** matjazp has joined #openstack-infra | 20:15 | |
| *** jcoufal has joined #openstack-infra | 20:15 | |
| *** chuckC has joined #openstack-infra | 20:16 | |
| phschwartz | anteaya: ty, not in a rush. Just letting you guys know I got the check gate to pass for it. | 20:17 |
| anteaya | phschwartz: cool, thanks a good first step | 20:17 |
| *** matjazp has quit IRC | 20:19 | |
| *** chianingwang_ has quit IRC | 20:20 | |
| *** sarob has joined #openstack-infra | 20:21 | |
| phschwartz | fungi: I will clean that up. Vim hates me and has stopped showing trailing white space :( | 20:23 |
| fungi | phschwartz: you should give it a talking to | 20:24 |
| phschwartz | fungi: I am about to slap it around a bit and rewrite my .vimrc it is pissing me off enough | 20:24 |
| *** annegentle_ has quit IRC | 20:25 | |
| phschwartz | fungi: as to the non-voting on the pecan gate tests. I agree. Just need to figure out the correct place to make it non-voting. | 20:27 |
| *** andreykurilin_ has joined #openstack-infra | 20:29 | |
| openstackgerrit | Philip Marc Schwartz proposed a change to openstack-infra/config: Creation of vinz project in the openstack-infra scheme. https://review.openstack.org/93953 | 20:30 |
| *** arnaud__ has joined #openstack-infra | 20:30 | |
| *** miqui has quit IRC | 20:32 | |
| fungi | phschwartz: where you set the unused vinz-python33 job non-voting which you didn't need (because you're not configured to run it) instead add the job name there. then leave the job in the check pipeline but remove it from the gate pipeline (nobody looks at non-voting job results after a change has merged, so they're just wasted resources there) | 20:33 |
| *** mwagner_lap has quit IRC | 20:34 | |
| *** julim has quit IRC | 20:34 | |
| fungi | phschwartz: that would have the effect of running the cross-test script for vinz non-voting on new changes proposed to pecan (but get an okay from one of the pecan ptls first too) | 20:34 |
| *** markmcclain has joined #openstack-infra | 20:34 | |
| *** wenlock has joined #openstack-infra | 20:35 | |
| openstackgerrit | Philip Marc Schwartz proposed a change to openstack-infra/config: Creation of vinz project in the openstack-infra scheme. https://review.openstack.org/93953 | 20:35 |
| phschwartz | that fixes it then, forgot to remove it from the gate. :( | 20:35 |
| phschwartz | not my day | 20:35 |
| *** jcoufal has quit IRC | 20:39 | |
| *** nati_ueno has joined #openstack-infra | 20:40 | |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/devstack-gate: Fix collection of *.failure files https://review.openstack.org/98885 | 20:43 |
| openstackgerrit | Adam Gandelman proposed a change to openstack-infra/config: Update apt cache before installing UCA keyring https://review.openstack.org/98886 | 20:44 |
| *** wenlock has quit IRC | 20:45 | |
| mriedem | fungi: clarkb: mordred: can we move this so we can debug some grenade failures that started showing up last week? https://review.openstack.org/#/c/98885/ | 20:45 |
| openstackgerrit | Andrew Melton proposed a change to openstack/requirements: Add idmapshift for Nova https://review.openstack.org/98888 | 20:46 |
| *** matjazp has joined #openstack-infra | 20:46 | |
| *** gyee has quit IRC | 20:46 | |
| *** sweston has joined #openstack-infra | 20:46 | |
| openstackgerrit | Andrew Melton proposed a change to openstack/requirements: Add idmapshift for Nova https://review.openstack.org/98888 | 20:47 |
| openstackgerrit | A change was merged to openstack-infra/os-loganalyze: construct syslog debug level https://review.openstack.org/97863 | 20:48 |
| fungi | mriedem: it'll be about 1.5 hours before that starts running check jobs though, so no actual way to observe the fix... but given the current backlog and need to press forward on the stability front i'm okay approving it blind | 20:48 |
| fungi | it at least looks correct and is quite trivial, so i'm not inclined to spend time testing it myself | 20:48 |
| mtreinish | fungi: and we know what's currently in tree doesn't work either :) | 20:49 |
| mriedem | fungi: sure. given how many times i've seen grenade fail with no services started since last week, it might even test itself | 20:49 |
| mtreinish | so it's not like it'll be worse | 20:49 |
| *** Sukhdev has joined #openstack-infra | 20:49 | |
| fungi | mriedem: all's the better | 20:49 |
| *** matjazp has quit IRC | 20:50 | |
| fungi | i'll let sdague approve and then i'm okay enqueuing+promoting it if we're worried we won't be able to iterate on it fast enough | 20:50 |
| fungi | though i need to start cooking dinner soonish | 20:51 |
| fungi | so will try to keep one eye on irc | 20:51 |
| lifeless | dhellmann: spec updated | 20:52 |
| *** e0ne has quit IRC | 20:53 | |
| lifeless | dhellmann: I note that oslo-specs isn't advertised on IRC? | 20:53 |
| *** fifieldt_ has quit IRC | 20:54 | |
| sdague | mriedem: +A | 20:56 |
| sdague | fungi: I'd also say that one is fair to promote, because it should help expose | 20:57 |
| mriedem | sdague: thanks | 20:57 |
| clarkb | aroo? sorry was eating lunch | 20:58 |
| dhellmann | lifeless: I think the bot config is there, but there was an issue with the bot dying last week. I don't know the current state. | 20:58 |
| sdague | so that ssh (possibly floating ip) failure is still there | 20:59 |
| *** duran has joined #openstack-infra | 21:00 | |
| sdague | just had something get bombed on it | 21:00 |
| *** marcoemorais has quit IRC | 21:00 | |
| *** marcoemorais has joined #openstack-infra | 21:01 | |
| *** Ryan_Lane1 has joined #openstack-infra | 21:01 | |
| lifeless | dhellmann: ack | 21:01 |
| *** Ryan_Lane has quit IRC | 21:03 | |
| *** andreykurilin__ has joined #openstack-infra | 21:04 | |
| *** andreykurilin_ has quit IRC | 21:04 | |
| *** fifieldt_ has joined #openstack-infra | 21:07 | |
| *** marcoemorais has quit IRC | 21:08 | |
| *** gyee has joined #openstack-infra | 21:08 | |
| *** marcoemorais has joined #openstack-infra | 21:08 | |
| *** godp1301 has quit IRC | 21:08 | |
| clarkb | mordred: are you still around? the unbound stuff is not really working the way I expect it to with your latest stack | 21:09 |
| *** duran has quit IRC | 21:10 | |
| mordred | clarkb: sup? | 21:10 |
| *** mbacchi has quit IRC | 21:10 | |
| *** morganfainberg_Z is now known as morganfainberg | 21:10 | |
| clarkb | mordred: I don't see where in the dib stuff you set install_resolv_conf to false | 21:10 |
| clarkb | mordred: and unbound updating my resolv.conf is failing dib builds | 21:11 |
| clarkb | mordred: was it just removed in a rebase somewhere? | 21:11 |
| *** praneshp has quit IRC | 21:11 | |
| clarkb | mordred: I think I can just hardcode install_resolv_conf => false in prepare-node | 21:11 |
| mordred | clarkb: hrm. it's entirely possible it got borked ina rebase | 21:11 |
| mordred | clarkb: ++ | 21:11 |
| clarkb | ok wanted to make sure I wasn't missing something | 21:12 |
| mordred | clarkb: did the database grants patch look ok to your brain? | 21:13 |
| Shrews | devananda: sdague: https://review.openstack.org/98201 finally passed testing. Would love to see you guys review that so we can push it through and avoid the ironic mess again. | 21:13 |
| clarkb | mordred: mostly. But it doesn't quite work. The mysql service breaks in dib | 21:13 |
| clarkb | mordred: also the slave_db.pp has requires on things that won't exist if you instantiate that file directly | 21:13 |
| clarkb | mordred: I am working towards testing and cleaning it up but have hit the unbound thing :) hope to have concrete useful feedback soon | 21:14 |
| *** arnaud__ has quit IRC | 21:14 | |
| mordred | clarkb: spectacular | 21:14 |
| *** sarob has quit IRC | 21:14 | |
| clarkb | mordred: though I am beginning to consider encoding all of the initial state setup in dib directly. | 21:15 |
| *** sarob has joined #openstack-infra | 21:15 | |
| clarkb | mordred: so have puppet install and configure mysql. Then dib in post-install.d can add users, edit perms, and add schemas | 21:15 |
| mordred | clarkb: I have also considered that | 21:15 |
| mordred | clarkb: I think the "make sure it's encoded in two different places" thing is the really hard part | 21:16 |
| clarkb | in this case we won't have to manage a split brain if we get all slaves on dib | 21:16 |
| mordred | ++ | 21:16 |
| *** sarob has quit IRC | 21:16 | |
| mordred | I mean, we'll have a split brain just for a transition period - but yes | 21:16 |
| *** lcostantino has joined #openstack-infra | 21:16 | |
| clarkb | right | 21:16 |
| *** mdenny has quit IRC | 21:17 | |
| clarkb | but I need to see how clunky this is with the puppet before I recommend ^ | 21:17 |
| mordred | clarkb: basically, we can just have puppet do openstack_project::template and then rewrite the slave config in dib, yeah? | 21:17 |
| mordred | clarkb: ++ | 21:17 |
| clarkb | mordred: no I don't want to rewrite the whole slave config yet :) | 21:17 |
| mordred | :) | 21:17 |
| clarkb | puppet is really good at installing things | 21:17 |
| clarkb | I think I want it to install mysql and postgres too | 21:17 |
| mordred | oh - I read what you said properly | 21:18 |
| mordred | clarkb: so, keep the split in the db patch - but just remove the slave_db.pp and rewrite _that_ in dib | 21:18 |
| clarkb | yup | 21:18 |
| *** smarcet has quit IRC | 21:20 | |
| clarkb | mordred: mostly because figuring out various config options and which package to install across distros and so on is in puppet and working and working well | 21:20 |
| clarkb | mordred: once we have that writing a make DB script is easy | 21:20 |
| *** wenlock has joined #openstack-infra | 21:23 | |
| clarkb | mordred: looks like you can't set mysql root password if the service is off... | 21:23 |
| clarkb | mordred: I think we can get away with that | 21:24 |
| *** mdenny has joined #openstack-infra | 21:25 | |
| *** arnaud__ has joined #openstack-infra | 21:26 | |
| clarkb | mordred: oh and I am going to push a change to go back to if in_config == 'true' because facter gives us a string not a boolean | 21:27 |
| clarkb | mordred: so setting that to 'false' in facter will still evaluate to true in puppet without the string comparison | 21:28 |
| *** mrda-longweekend is now known as mrda | 21:30 | |
| *** lcheng has quit IRC | 21:30 | |
| *** gyee has quit IRC | 21:31 | |
| openstackgerrit | Adam Gandelman proposed a change to openstack-infra/elastic-recheck: Add query for Ironic bug 1328274 https://review.openstack.org/98905 | 21:32 |
| uvirtbot | Launchpad bug 1328274 in ironic "devstack jobs failing due to n-cpu issues updating resources from conductor" [Undecided,New] https://launchpad.net/bugs/1328274 | 21:32 |
| *** gondoi is now known as zz_gondoi | 21:33 | |
| *** dkranz has quit IRC | 21:34 | |
| openstackgerrit | Elizabeth K. Joseph proposed a change to openstack-infra/publications: Add instructions for using template https://review.openstack.org/98906 | 21:34 |
| pleia2 | (officially not explaining that again :)) | 21:35 |
| *** zz_gondoi is now known as gondoi | 21:35 | |
| *** eharney has quit IRC | 21:35 | |
| *** lcheng has joined #openstack-infra | 21:35 | |
| *** bauzas has quit IRC | 21:37 | |
| *** arnaud__ has quit IRC | 21:37 | |
| *** slamont has quit IRC | 21:40 | |
| openstackgerrit | Andrew Melton proposed a change to openstack/requirements: Add idmapshift for Nova https://review.openstack.org/98888 | 21:41 |
| *** dkliban is now known as dkliban_afk | 21:41 | |
| mordred | pleia2: did yuo see my latest crack? https://review.openstack.org/98670 | 21:41 |
| *** slamont has joined #openstack-infra | 21:42 | |
| *** signed8bit has quit IRC | 21:42 | |
| pleia2 | mordred: yeah, I actually just mentioned that in an hp email "but our template is still slidy right now" ;) | 21:43 |
| mordred | pleia2: I haven't figured out how to hook the zuul demo into the reveal slide advancement thing | 21:43 |
| mordred | pleia2: but something tells me someone like krotscheck could figure it out in like 4 seconds | 21:44 |
| pleia2 | mordred: ah yeah, web people do magic | 21:44 |
| *** wenlock has quit IRC | 21:44 | |
| mordred | pleia2: I particularly like the vertical slide serieses - they make for a fun organization and an easy way to have drill-down slides or to just show an overview slide | 21:44 |
| pleia2 | I'm also working on updating some of the screenshots in the overview, new gerrit and zuul page is really old | 21:44 |
| mordred | ++ | 21:45 |
| anteaya | I'm following pleia2's advice, you change too fast mordred | 21:45 |
| anteaya | don't get me wrong you are very entertaining | 21:45 |
| anteaya | but too mecurial for me to tie a horse to | 21:45 |
| mordred | pleia2: well, we should maybe land the first patch of mine from the overview branch | 21:45 |
| mordred | pleia2: I did several updates in there | 21:45 |
| *** mriedem has quit IRC | 21:45 | |
| pleia2 | mordred: or do the template branch first? | 21:46 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: Setup a private gerrit instance for security reviews https://review.openstack.org/47937 | 21:46 |
| mordred | pleia2: either way - just calling out that we've potentially got some fun merging coming up | 21:46 |
| pleia2 | mordred: yeah | 21:46 |
| *** mrodden has quit IRC | 21:46 | |
| *** matjazp has joined #openstack-infra | 21:47 | |
| clarkb | mordred: mv: cannot create hard link `/tmp/image.2l9ceN8b/mnt/var/lib/postgresql/9.1/main/pg_stat_tmp/pgstat.stat' to `/tmp/image.2l9ceN8b/mnt/lib/modules/3.2.0-64-generic/kernel/drivers/video/nvidia': Operation not permitted | 21:48 |
| clarkb | mordred: I have no idea why that is happening but I got puppet to run without creating any of the dbs and users | 21:48 |
| *** signed8bit has joined #openstack-infra | 21:48 | |
| clarkb | mordred: but no image due to ^ | 21:48 |
| clarkb | this is troubling for a few reasons. first we don't need nvidia drivers so o_O. second why is it hardlinking those two things together? Is this something you have seen before? | 21:49 |
| *** praneshp has joined #openstack-infra | 21:50 | |
| *** alkari has quit IRC | 21:50 | |
| mordred | I have not seen that | 21:50 |
| mordred | lifeless: ^^ why does postgres want nvidia? | 21:50 |
| *** alkari has joined #openstack-infra | 21:51 | |
| *** matjazp has quit IRC | 21:51 | |
| *** jerryz has joined #openstack-infra | 21:54 | |
| fungi | uh woa | 21:55 |
| lifeless | mordred: wait what | 21:55 |
| lifeless | mordred: thats very very very very very wrong | 21:55 |
| fungi | hardlink pgstat.stat to the nvidia lkm driver directory? | 21:55 |
| clarkb | ya | 21:55 |
| clarkb | I have no idea why that happened. running again to see if it is reproduceable | 21:55 |
| *** _nadya_ has joined #openstack-infra | 21:56 | |
| * fungi picked the wrong time to look at irc with food in mouth | 21:56 | |
| clarkb | dib did correctly fail so not worried about weirdness getting onto slaves. just trying to build images | 21:56 |
| fungi | this almost looks like some sort of subtle backdoor hack | 21:57 |
| *** andreaf has joined #openstack-infra | 21:57 | |
| fungi | "grant the postgres daemon the ability to write kernel drivers" | 21:57 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: Setup a private gerrit instance for security reviews https://review.openstack.org/47937 | 21:57 |
| fungi | more likely inodes got screwed up in the filesystem table | 21:58 |
| fungi | but still a little scary | 21:58 |
| clarkb | http://paste.openstack.org/show/83452/ | 21:58 |
| clarkb | next run just ran into sha256sum: WARNING: 1 computed checksum did NOT match | 21:58 |
| *** andreykurilin__ has quit IRC | 21:58 | |
| clarkb | so ya I think I may have fs corruption | 21:58 |
| * clarkb tries again | 21:59 | |
| mordred | lifeless: your tool is breaking clarkb's filesystem | 21:59 |
| clarkb | hrm that is consistent now | 21:59 |
| clarkb | time to clear cache | 21:59 |
| *** rlandy_ has quit IRC | 21:59 | |
| clarkb | rm'ing things in cache got beyond the hash check | 22:00 |
| *** dims_ has joined #openstack-infra | 22:00 | |
| * fungi is now less frightened | 22:00 | |
| *** _nadya_ has quit IRC | 22:00 | |
| *** UtahDave has quit IRC | 22:01 | |
| *** markmcclain has quit IRC | 22:01 | |
| *** james_li has quit IRC | 22:02 | |
| *** gyee has joined #openstack-infra | 22:02 | |
| *** adalbas has quit IRC | 22:02 | |
| *** dims has quit IRC | 22:02 | |
| *** ramashri has quit IRC | 22:05 | |
| *** ramashri has joined #openstack-infra | 22:05 | |
| lifeless | mordred: yeah no. | 22:05 |
| ianw | fungi: looks like it's some gpu accelerated db thing -> https://wiki.postgresql.org/wiki/PGStrom? | 22:06 |
| ianw | fungi: agree, first thought was "rootkit" :) | 22:06 |
| *** gondoi is now known as zz_gondoi | 22:06 | |
| clarkb | oh neat except hrm | 22:07 |
| ianw | oh, fs corruption is even more likely | 22:08 |
| clarkb | ianw: yeah it may be fs corruption | 22:08 |
| ianw | there's no other references to such a thing i can find | 22:08 |
| *** nati_ueno has quit IRC | 22:08 | |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Fix collection of *.failure files https://review.openstack.org/98885 | 22:08 |
| clarkb | the current run is puppeting now should get a good idea of whether or not this is consistent from that | 22:09 |
| clarkb | I am going to start putting my patches up into gerrit so that someone else can try to independently test as well | 22:09 |
| clarkb | mordred: ^ | 22:09 |
| fungi | ianw: any sufficiently advanced rootkit is indistinguishable from random filesystem corruption (apologies to arthur c. clarke) | 22:09 |
| mordred | clarkb: awesome | 22:09 |
| *** marcoemorais has quit IRC | 22:09 | |
| *** marcoemorais has joined #openstack-infra | 22:10 | |
| *** ihrachyshka has quit IRC | 22:10 | |
| *** isviridov_ has joined #openstack-infra | 22:10 | |
| *** salv-orlando has quit IRC | 22:10 | |
| *** marcoemorais has quit IRC | 22:11 | |
| *** marcoemorais has joined #openstack-infra | 22:11 | |
| jogo | fungi: http://logs.openstack.org/98/97598/1/check/check-grenade-dsvm-partial-ncpu/2125f27/console.html#_2014-06-09_19_16_59_231 | 22:11 |
| *** marcoemorais has quit IRC | 22:12 | |
| *** marcoemorais has joined #openstack-infra | 22:13 | |
| sdague | clarkb: do you have to tell the logstash workers which files have INFO filters on them? | 22:14 |
| *** salv-orlando has joined #openstack-infra | 22:14 | |
| *** salv-orlando has quit IRC | 22:15 | |
| clarkb | sdague: no, it applies the filter to everything | 22:16 |
| *** salv-orlando has joined #openstack-infra | 22:16 | |
| *** isviridov has quit IRC | 22:17 | |
| *** salv-orlando has quit IRC | 22:17 | |
| sdague | cool | 22:17 |
| sdague | clarkb: this would be a nice land as well - https://review.openstack.org/#/c/98867/ | 22:18 |
| fungi | jogo: "Raised if a server responds with a HTTP status code that we don’t understand" | 22:18 |
| fungi | jogo: i'll check the apache error logs | 22:18 |
| sdague | then we'd have linking in syslogs | 22:18 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add fingerprint fur bug 1268725 https://review.openstack.org/98913 | 22:19 |
| uvirtbot | Launchpad bug 1268725 in openstack-ci "ConnectionError: HTTPConnectionPool(host='pypi.openstack.org', port=80): Max retries exceeded" [Medium,Triaged] https://launchpad.net/bugs/1268725 | 22:19 |
| *** salv-orlando has joined #openstack-infra | 22:19 | |
| jogo | fungi: looks like you filed a bug for that a while back | 22:19 |
| *** salv-orlando has quit IRC | 22:20 | |
| *** otherwiseguy has quit IRC | 22:20 | |
| *** miqui has joined #openstack-infra | 22:20 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Split the database grants into their own class https://review.openstack.org/98586 | 22:21 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Add support for disk-image-builder in nodepool https://review.openstack.org/88479 | 22:21 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Add additional node types to dib elements https://review.openstack.org/90565 | 22:21 |
| clarkb | mordred: ^ those are the changes I am currently working iwth | 22:21 |
| clarkb | assuming I actually applied the diffs properly | 22:21 |
| clarkb | mordred: note the changes to not manage mysql service, not set root db passwords, rename node-bare to node-thick, and check in_chroot as a string not boolean value | 22:22 |
| clarkb | mordred: oh and set the don't install resolv conf flag | 22:22 |
| *** salv-orlando has joined #openstack-infra | 22:22 | |
| fungi | jogo: hm... so i find absolutely no reference to that get in the error log/ | 22:23 |
| clarkb | I need to pop out shortly to pick up new glasses \o/ | 22:23 |
| *** salv-orlando has quit IRC | 22:23 | |
| clarkb | waiting for current dib run to do its thing before I disappear though | 22:23 |
| *** flaper87 is now known as flaper87|afk | 22:23 | |
| *** mwagner_lap has joined #openstack-infra | 22:24 | |
| *** krotscheck has quit IRC | 22:24 | |
| *** krotscheck has joined #openstack-infra | 22:25 | |
| clarkb | woot it is qemu image converting now | 22:25 |
| clarkb | mordred: can you grab those changes and dib node-think instead of node-devstack? | 22:25 |
| jogo | fungi: perahps a network issue | 22:25 |
| *** salv-orlando has joined #openstack-infra | 22:25 | |
| clarkb | mordred: and independently verify this mostly works | 22:26 |
| fungi | jogo: no mention of it in the access log either. i suspect hpcloud might put a layer 7 proxy in front of http requests... anybody knowing internals of hpcloud confirm? | 22:26 |
| clarkb | fungi: pretty sure there is fancy load balancer there | 22:26 |
| *** marcoemorais has quit IRC | 22:26 | |
| mordred | clarkb: I can - but I'm about to run out the door to a doctor's appointment | 22:26 |
| clarkb | mordred: no rush then | 22:26 |
| *** marcoemorais has joined #openstack-infra | 22:26 | |
| anteaya | mordred: be well | 22:27 |
| fungi | i can't imaging a "network problem" causing a slave in hpcloud to get an "unrecognized http status code" from a server in rackspace because of a "network issue" | 22:27 |
| *** marcoemorais has quit IRC | 22:27 | |
| fungi | er, splicing sentences in my brain again | 22:27 |
| *** marcoemorais has joined #openstack-infra | 22:27 | |
| fungi | and typing horribly | 22:27 |
| fungi | i should just officially start my evening | 22:27 |
| *** michchap has quit IRC | 22:28 | |
| mordred | anteaya: regular stuff- nothing big | 22:29 |
| anteaya | glad to hear it | 22:29 |
| *** weshay has quit IRC | 22:29 | |
| *** matjazp has joined #openstack-infra | 22:29 | |
| *** sweston has quit IRC | 22:30 | |
| *** matjazp has quit IRC | 22:30 | |
| *** mfer has quit IRC | 22:30 | |
| *** cody-somerville has joined #openstack-infra | 22:31 | |
| *** cody-somerville has quit IRC | 22:31 | |
| *** cody-somerville has joined #openstack-infra | 22:31 | |
| *** camunoz has joined #openstack-infra | 22:36 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: Setup a private gerrit instance for security reviews https://review.openstack.org/47937 | 22:36 |
| *** rainya has quit IRC | 22:37 | |
| *** freyes has quit IRC | 22:37 | |
| fungi | zaro: awesome! i need to find time to look at that ^ between meetings tomorrow | 22:38 |
| *** cody-somerville has quit IRC | 22:38 | |
| *** chianingwang has quit IRC | 22:38 | |
| *** rainya has joined #openstack-infra | 22:43 | |
| *** hemna is now known as hemna_ | 22:45 | |
| *** jgrimm has quit IRC | 22:45 | |
| *** sballe_ has joined #openstack-infra | 22:46 | |
| *** jhesketh has quit IRC | 22:46 | |
| *** nati_ueno has joined #openstack-infra | 22:47 | |
| *** jamielennox|away is now known as jamielennox | 22:47 | |
| *** sballe has quit IRC | 22:47 | |
| *** dangers is now known as dangers_away | 22:47 | |
| *** freyes has joined #openstack-infra | 22:48 | |
| *** crc32 has quit IRC | 22:48 | |
| *** Sukhdev has quit IRC | 22:49 | |
| *** aysyd has quit IRC | 22:51 | |
| *** doude has quit IRC | 22:53 | |
| *** doude has joined #openstack-infra | 22:53 | |
| *** crc32 has joined #openstack-infra | 22:54 | |
| *** locke105 has quit IRC | 22:56 | |
| sdague | clarkb: any update on getting some more logstash workers cranking? | 22:59 |
| *** freyes has quit IRC | 22:59 | |
| mattoliverau | Morning everyone! | 22:59 |
| clarkb | sdague none sorry. I am afk now to get glasses | 23:02 |
| clarkb | I can start spinning up nodes later this evening | 23:03 |
| sdague | ok, the other thing, before I call it a night, the gerrit reset votes on commit change? | 23:04 |
| sdague | is that something there is a review for already? | 23:04 |
| clarkb | I have no idea where that is | 23:04 |
| sdague | I'm definitely getting massively broken by the current behavior | 23:04 |
| clarkb | other people were working it I think | 23:04 |
| *** MarkAtwood has quit IRC | 23:04 | |
| sdague | ok, I'll poke in the morning | 23:04 |
| clarkb | fungi ^ was that you? | 23:05 |
| clarkb | iirc it required manual acl update | 23:05 |
| *** amcrn has joined #openstack-infra | 23:13 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Remove duplicate query for 1326820 https://review.openstack.org/98919 | 23:13 |
| *** zzelle_ has quit IRC | 23:18 | |
| *** virmitio has quit IRC | 23:18 | |
| *** locke105 has joined #openstack-infra | 23:19 | |
| fungi | clarkb: sdague: oh, it requires a push of an all-projects config (acl) update and then hopefully not a restart | 23:19 |
| fungi | i was hoping zaro would weigh in when i was asking about it last week in case there's something i'm missing to adjust it in the webui | 23:20 |
| fungi | but for whatever reason it looks like those options may only be switchable by setting them in the acl file directly, and not at all implemented as a setting in the web interface | 23:21 |
| *** nati_ueno has quit IRC | 23:22 | |
| *** atiwari has quit IRC | 23:24 | |
| zaro | missed that fungi. but i think changes to all-projects config does not require gerrit restart. | 23:25 |
| *** zzelle has joined #openstack-infra | 23:25 | |
| *** morganfainberg is now known as morganfainberg_Z | 23:25 | |
| *** zzelle has quit IRC | 23:25 | |
| fungi | zaro: that's a plus at least. any feeling for why the switches to control resetting votes on rebases or commit message changes aren't exposed in the webui for editing the all-projects acl? | 23:26 |
| zaro | fungi: i have no idea. some things are exposed while others can only be done by pushing all-projects.config i think labels is another thing that can only be configured in project config files. | 23:27 |
| jogo | do we have any way of tracking how many gate jobs get invalidated? | 23:28 |
| *** nati_ueno has joined #openstack-infra | 23:28 | |
| *** jhesketh has joined #openstack-infra | 23:29 | |
| *** zns has quit IRC | 23:30 | |
| jhesketh | Morning | 23:30 |
| *** nati_uen_ has joined #openstack-infra | 23:31 | |
| fungi | zaro: okay, cool. just making sure that was reasonable/normal | 23:31 |
| fungi | jogo: #define "invalidated" | 23:31 |
| jogo | fungi: thrown out | 23:32 |
| *** nati_ueno has quit IRC | 23:32 | |
| jogo | as I have a hunch we are throwing out more then usual now, which is why we we are using all the cloud | 23:32 |
| fungi | aborted/cancelled because of a new patchset upload or gate reset ahead of them prior to completing the job? | 23:32 |
| jogo | gate reset | 23:32 |
| fungi | we might have graphite counts on started and success/failure, which could yield a difference... nothing cooked though afaik | 23:33 |
| jogo | ahh thanks | 23:35 |
| jogo | in other news ceilometer unit tests fail 16% of the time in the gate | 23:35 |
| fungi | that's an entirely unsurprising statistic considering what i've seen of them in the gate | 23:36 |
| devananda | hi! i'm seeing many *-tempest-dsvm*ironic* tests failing due to what seem to me to be network failures | 23:37 |
| devananda | is this a known issue right now? | 23:37 |
| jogo | ohhh that goes to 18% in a week | 23:37 |
| *** sweston has joined #openstack-infra | 23:38 | |
| zaro | jhesketh: hey. | 23:40 |
| jhesketh | Hello zaro! | 23:41 |
| anteaya | morning jhesketh | 23:41 |
| jhesketh | Morning anteaya | 23:41 |
| zaro | jhesketh: unfortunately i don't think breaking up jjb is gonna work for hipster use case. | 23:42 |
| *** morganfainberg_Z is now known as morganfainberg | 23:42 | |
| *** alkari has joined #openstack-infra | 23:42 | |
| *** masayukig_ has quit IRC | 23:43 | |
| jhesketh | hmm, interesting, why's that zaro? | 23:43 |
| *** masayukig has joined #openstack-infra | 23:43 | |
| zaro | jhesketh: jjb is designed to inject xml into it's data structure. All of the jjb modules (like publisher, wrapper, reporters, etc..) create xml directly. it's never stored in any other data structure. | 23:44 |
| jhesketh | yeah, that was the challenge I faced when I looked at it.. that's why I worked around it by converting xml to dict | 23:45 |
| jhesketh | however jeblair thought it was possible, so I'd like to get his thoughts | 23:45 |
| anteaya | morning mattoliverau, I am reading backwards | 23:45 |
| mattoliverau | anteaya: lol, morning. Have a good weekend? | 23:45 |
| zaro | jhesketh: hmm, i'm not seeing an easy way to do it without rewritting all of those modules. | 23:46 |
| jhesketh | zaro: is it possible to expand all of the templates/groups/projects etc into individual jobs before passing off to the modules? | 23:46 |
| anteaya | mattoliverau: I did actually | 23:47 |
| anteaya | mattoliverau: saw a cow and calf moose and stared a long time at a wolf | 23:47 |
| anteaya | we looked at each other for like 5 minutes before the wolf moved off and I saw its tail | 23:48 |
| zaro | jhesketh: it doesn't seem like it to me because the modules are the thing that are creating the jobs. | 23:48 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Remove duplicate query for 1326820 and 1326855 https://review.openstack.org/98919 | 23:48 |
| *** sweston has quit IRC | 23:49 | |
| *** homeless has quit IRC | 23:50 | |
| jhesketh | zaro: hmm... might need to consider a large refactor | 23:50 |
| jhesketh | or something new in TH that just interprets the files correctly | 23:50 |
| adam_g | has anything changed to image builds recently that would cause this to pop up on HP slaves but not RAX? https://bugs.launchpad.net/tripleo/+bug/1326754 | 23:51 |
| uvirtbot | Launchpad bug 1326754 in tripleo "Ubuntu CI failures : Unable to locate package ubuntu-cloud-keyring" [High,In progress] | 23:51 |
| fungi | adam_g: we use the sources lists provided on the base images, which aren't created by ubuntu but rather by the providers. my guess would be that hp does not add it to their images but rax does | 23:53 |
| fungi | adam_g: actually, i think the images we're using in hpcloud are actual ubuntu-provided images now (as of about a month ago), but that rax still rolls their own | 23:53 |
| *** lcheng has quit IRC | 23:54 | |
| fungi | adam_g: we do a lot of fiddling already to deal with the differences between base images in various providers. hopefully now that we can start using glance in hp and rax that will subside soon | 23:55 |
| adam_g | fungi, yeah.. the package is on the mirrors referenced by the HP nodes. i wonder if the HP nodes are coming up with a completely cold apt cache. | 23:55 |
| zaro | jhesketh: i think it would need to be large refactor. it would require making generic parts of jjb (templates, jobs, project, macros etc..) agnostic to the data structure used in derived tools so that jenkins part can use xml and hipster part can use dict. | 23:56 |
| fungi | adam_g: oh, possible... though devstack should retry to apt-get update of an apt-get install fails | 23:56 |
| fungi | s/of/if/ | 23:56 |
| adam_g | fungi, the errors happening outside of devstack itself. | 23:56 |
| adam_g | https://review.openstack.org/#/c/98886/ should help if that is the issue | 23:57 |
| mattoliverau | anteaya: wow! | 23:57 |
| fungi | adam_g: oh, looking | 23:57 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!