| fungi | does it open a new connection per query or something? | 00:00 |
|---|---|---|
| clarkb | it possible it may | 00:00 |
| clarkb | it uses sqlalchemy session objects. not sure if it creats a new one per query but can probably find out | 00:00 |
| comstud | jogo: If you can get me a list of instance uuids and regions | 00:01 |
| comstud | i don't have a lot of time right now | 00:01 |
| jogo | comstud: follow me to -infra | 00:01 |
| comstud | i'm in infra | 00:02 |
| jogo | oh woops hehe | 00:02 |
| comstud | here | 00:02 |
| jogo | fungi: ^^ | 00:02 |
| comstud | hahaha | 00:02 |
| jogo | clarkb: ^ | 00:02 |
| clarkb | oh yup it uses a session factory so that may be it | 00:02 |
| jogo | comstud graciously accepted being volunteered to clean out some deleting,error instances | 00:02 |
| anteaya | ha ha ha | 00:03 |
| comstud | i'm going to attempt to delegate it | 00:03 |
| comstud | but if i can't wake someone up, i'll try to take care of it quickly | 00:03 |
| clarkb | phschwartz was the one looking at it and I thought he had a list | 00:03 |
| comstud | oh okay | 00:03 |
| clarkb | he was looking at it from the rax side | 00:03 |
| jogo | clarkb: comstud is rax too | 00:04 |
| comstud | perfect, i don't need to do anything | 00:04 |
| anteaya | clarkb: he is at a concert | 00:04 |
| clarkb | it was easy to determine which nodes were derpy bceause cell and nova api weren't coordinating with each other or some such | 00:04 |
| clarkb | anteaya: right just pointing out that we shouldn't need to give anyone data | 00:04 |
| clarkb | they have it | 00:04 |
| anteaya | ah | 00:04 |
| jogo | clarkb: true but the person with data is AFK | 00:04 |
| jogo | and we can get someone to help now | 00:04 |
| clarkb | jogo: right but I don't have the data | 00:05 |
| clarkb | jogo: I only ave data for what nova told me which isn't correct or some such | 00:05 |
| jogo | clarkb: oh can't you just do a nova list and see | 00:05 |
| jogo | clarkb: ohh I thought any instance in error,deleting is bad | 00:05 |
| anteaya | what about fungi's pastes | 00:06 |
| clarkb | fungi: the scoped session it uses should be one per thread | 00:06 |
| clarkb | fungi: and we use a one thread server so we should be ko | 00:06 |
| anteaya | http://paste.openstack.org/show/83880/ | 00:06 |
| clarkb | fungi: clearly we arent' so something is up | 00:06 |
| anteaya | http://paste.openstack.org/show/83881/ | 00:06 |
| anteaya | http://paste.openstack.org/show/83882/ | 00:06 |
| anteaya | http://paste.openstack.org/show/83885/ | 00:06 |
| clarkb | anteaya: that shows you what the states are according to nodepool | 00:06 |
| clarkb | but aiui this is a cells thing and we have no access to cells info | 00:06 |
| anteaya | there are uuid's are there not? | 00:06 |
| *** plars has joined #openstack-infra | 00:07 | |
| clarkb | anteaya: there are, but some of those nodes may be fine | 00:07 |
| clarkb | we have no way of knowing | 00:07 |
| clarkb | unless I misunderstood what was said earlier | 00:07 |
| comstud | this is not a cells thing | 00:07 |
| anteaya | they are in error status | 00:07 |
| comstud | at least, i have no idea what that means in this context | 00:07 |
| anteaya | that might be an incorrect status | 00:07 |
| anteaya | :( | 00:07 |
| comstud | nova ignores deletes on instances in task_state deleting. | 00:07 |
| clarkb | comstud: cells says I am ignoring you nova api | 00:07 |
| comstud | is the problem | 00:07 |
| clarkb | nova api says no really please do this | 00:08 |
| comstud | clarkb: for 'delete' or something else? | 00:08 |
| comstud | it's a bug in compute/api | 00:08 |
| clarkb | comstud: for delete because nova sometimes drops delets on teh floor | 00:08 |
| jogo | clarkb: comstud wrote cells | 00:08 |
| clarkb | jogo: I know | 00:08 |
| clarkb | so we reissue delete requests | 00:08 |
| comstud | there's the same behavior with cells and without cells in this case | 00:08 |
| clarkb | comstud: ah ok | 00:09 |
| comstud | is what i'm trying to say | 00:09 |
| comstud | it's a general 'bug' in nova | 00:09 |
| clarkb | it was framed as a cells thing earlier | 00:09 |
| comstud | jogo put up a review | 00:09 |
| clarkb | but can totally believe its a noav bug | 00:09 |
| comstud | yeah, that's quite possible... | 00:09 |
| comstud | cells can be blamed for a lot of things | 00:09 |
| comstud | :) | 00:09 |
| jogo | clarkb: yeah I didn't see any cells specific thing in there yet | 00:09 |
| clarkb | anyways we have no way of knowing which is why we reissue the deletes | 00:09 |
| comstud | yeah | 00:09 |
| comstud | deletes in 'deleting' task_state | 00:09 |
| comstud | are ignored by nova | 00:09 |
| clarkb | I can give you a complete list of the things we think should be deleting | 00:10 |
| comstud | so we have to reset the task_state to fix them | 00:10 |
| jogo | clarkb: the error is happening on the cell level but its not a cells isssue | 00:10 |
| comstud | jogo: This is why i think it's kinda dumb to leave things in 'deleting' | 00:10 |
| comstud | because it's a lie | 00:10 |
| * jogo owes comstud a beer | 00:10 | |
| comstud | jogo: correct | 00:10 |
| clarkb | comstud: would a complete list of things we think should be deleting help? | 00:10 |
| comstud | jogo: and: also correct | 00:10 |
| comstud | :) | 00:10 |
| jogo | comstud: at least they have an error state. but I am not disagreeing with you | 00:10 |
| clarkb | from a nova api consumer perspective? | 00:10 |
| comstud | clarkb: Yes.. a list of instance uuids and their region | 00:10 |
| comstud | optionally.. just your tenant ids | 00:11 |
| comstud | or id | 00:11 |
| comstud | we probably have that :) | 00:11 |
| comstud | but | 00:11 |
| comstud | i don't want to just blanket reset things in error/deleting | 00:11 |
| comstud | i'd rather you tell me what you want fixed | 00:11 |
| comstud | actually, it sounds like someone is going to kick it off here anyway | 00:12 |
| *** melwitt has quit IRC | 00:13 | |
| *** ramashri has joined #openstack-infra | 00:13 | |
| comstud | on some tenant id that ends with the same number that it starts with. | 00:13 |
| comstud | :) | 00:13 |
| clarkb | well I almost have the list if I could convince pastebinit to stop dying | 00:14 |
| comstud | hehe | 00:15 |
| *** W00die has quit IRC | 00:15 | |
| anteaya | get MySQL to stick around | 00:15 |
| markmcclain | from the traceback..the original error that trigger the nova bug was related to contacting neutron? | 00:15 |
| comstud | clarkb: Would you prefer we also just delete them for you? | 00:16 |
| *** ianw has quit IRC | 00:16 | |
| comstud | after resetting task state? | 00:16 |
| markmcclain | networking failure or possible neutron problem? | 00:16 |
| comstud | heh | 00:16 |
| comstud | something. | 00:16 |
| *** ianw has joined #openstack-infra | 00:16 | |
| comstud | :) | 00:16 |
| comstud | for transparency purposes.. I think it's a scaling issue | 00:16 |
| comstud | with neutron | 00:17 |
| comstud | that we're working on... but TBH, I'm not 100% sure.. been busy on other things | 00:17 |
| jgriffith | clarkb: a start at any rate: https://review.openstack.org/99809 | 00:18 |
| markmcclain | comstud: thanks for sharing | 00:19 |
| *** matsuhashi has joined #openstack-infra | 00:19 | |
| clarkb | gah and of course xclip and firefox hate aech other | 00:19 |
| jogo | markmcclain: isn't neutron fun | 00:20 |
| comstud | haha | 00:20 |
| comstud | so, I actually gotta run | 00:21 |
| comstud | you can email me and I can forward it | 00:21 |
| anteaya | thanks comstud | 00:21 |
| comstud | but | 00:21 |
| comstud | it sounds like someone is actually working on this already | 00:21 |
| markmcclain | jogo: yep :) the tricky part is the backend heavily influences scaling | 00:21 |
| jogo | comstud: thanks! | 00:21 |
| comstud | by matching 'error'/'deleting' and your project id | 00:21 |
| comstud | and resetting | 00:21 |
| jogo | markmcclain: heh yeah | 00:21 |
| comstud | It also sounded like we were going to delete those for you afterwards | 00:22 |
| comstud | but anyway | 00:22 |
| clarkb | comstud: http://paste.openstack.org/show/83900/ | 00:22 |
| clarkb | woo xclip -selection clipboard | 00:22 |
| comstud | ok | 00:22 |
| jogo | clarkb: I use pastebinit | 00:22 |
| clarkb | jogo: it wouldn't work | 00:22 |
| clarkb | jogo: pastebinit -b http://paste.openstack.org -i foo borked | 00:23 |
| clarkb | could be related to the 500 errors | 00:23 |
| comstud | what's with the NULL server ids? | 00:23 |
| comstud | that seems... unpossible. | 00:23 |
| clarkb | comstud: I think that means nova didn't give us a uuid back | 00:23 |
| comstud | :) | 00:23 |
| comstud | wat | 00:23 |
| clarkb | either that or I raced against the db | 00:23 |
| clarkb | but they are >11 hours old so probably not that | 00:24 |
| jogo | anteaya: how does this look https://review.openstack.org/99796 | 00:24 |
| comstud | yeah, something seems wrong with that data | 00:24 |
| comstud | but | 00:25 |
| comstud | we'll ignore those | 00:25 |
| clarkb | ya thats actually amazing | 00:25 |
| clarkb | could be amazing from your side or amazing from our side | 00:25 |
| clarkb | we do have an alien lister we can use to hopefulyl sort that out | 00:25 |
| * clarkb tries this | 00:25 | |
| clarkb | I have zero nodes in the rax iad alien list | 00:26 |
| fungi | there are times when nova boot calls never get far enough to return an instance uuid | 00:27 |
| clarkb | we are doing about 200kb per half hour of es slow log | 00:29 |
| clarkb | we should be safe to leave that in place for a while | 00:29 |
| clarkb | fungi: I am going to put a small shell script in my homedir to turn off the slow log though | 00:29 |
| *** sweston has quit IRC | 00:29 | |
| clarkb | fungi: er homedir on es01 that way if you need it it is there | 00:29 |
| *** mrmartin has quit IRC | 00:30 | |
| *** thuc_ has quit IRC | 00:30 | |
| *** thuc has joined #openstack-infra | 00:31 | |
| comstud | ok, people will be working this | 00:32 |
| comstud | I gotta run | 00:32 |
| comstud | i'll bbl | 00:32 |
| clarkb | comstud: thanks | 00:33 |
| clarkb | fungi: two scripts, one to turn it on and the other to turn it off. | 00:33 |
| clarkb | fungi: pretty simple but this way you don't have to dig through docs | 00:33 |
| anteaya | jogo: I get it now, I think | 00:33 |
| fungi | clarkb: thanks for the heads up. SergeyLukjanov ^ that may be of interest to you too if elasticsearch01.o.o say... runs out of space for logs or something | 00:35 |
| *** thuc has quit IRC | 00:35 | |
| jogo | anteaya: cool, let me know if you have any other questsions | 00:35 |
| *** mmaglana has joined #openstack-infra | 00:36 | |
| anteaya | ahhhh, when can those deleting nodes get deleted | 00:37 |
| anteaya | that's about my only other question atm | 00:37 |
| anteaya | :D | 00:37 |
| anteaya | but I appreciate the walk though the status, I didn't have a clue how that was handled before | 00:38 |
| *** dims_ has joined #openstack-infra | 00:39 | |
| anteaya | polls close in 17 minutes and the thunder is threatening to take out my power | 00:43 |
| anteaya | boo | 00:43 |
| clarkb | anteaya: you guys don't vote by mail? | 00:44 |
| anteaya | I don't know if it is an option | 00:44 |
| anteaya | went into the legion was second in line this morning and first at my polling station | 00:44 |
| anteaya | the woman in front was 91 and brought her flashlight so she didn't have to ask for help to see the ballot | 00:45 |
| anteaya | feisty | 00:45 |
| anteaya | if I was out of the country I would do a mail in ballot or find out about it | 00:46 |
| anteaya | but mostly we show up, take the paper, mark it with a pencil, show the initals on the outside of the ballots to the returning officer and put it in the box | 00:46 |
| anteaya | s/outside of the ballots/outside of the ballot | 00:47 |
| *** jhesketh has quit IRC | 00:48 | |
| *** saper has joined #openstack-infra | 00:51 | |
| *** csheedy has joined #openstack-infra | 00:51 | |
| *** yamahata has joined #openstack-infra | 00:53 | |
| *** ramashri has quit IRC | 00:55 | |
| *** csheedy has quit IRC | 00:57 | |
| *** malini1 has joined #openstack-infra | 01:00 | |
| *** xchu has joined #openstack-infra | 01:01 | |
| *** mriedem has joined #openstack-infra | 01:03 | |
| *** nati_ueno has quit IRC | 01:04 | |
| clarkb | so I just realized I arrive in frankfurt on the day of the world cup final | 01:04 |
| anteaya | so it is a bad day to travel or a good day to arrive in germany? | 01:06 |
| anteaya | the sunday? | 01:06 |
| *** jhesketh has joined #openstack-infra | 01:07 | |
| anteaya | any world cup is a great time to be in toronto | 01:07 |
| anteaya | some part of the city will be celbrating | 01:07 |
| anteaya | and everyone else is invited to the party | 01:07 |
| tchaypo | unless they're rioting | 01:07 |
| clarkb | anteaya: I am guessing a little bit of both | 01:07 |
| clarkb | also I am sure the festivities will change quite a bit depending on who is playing | 01:07 |
| clarkb | anteaya: yes the sunday | 01:08 |
| anteaya | cool | 01:08 |
| anteaya | yes | 01:08 |
| anteaya | tchaypo: yeah, I can't remember world cup riots in toronto | 01:08 |
| mattoliverau | I think Josh and I are planning on arriving on the Saturday.. to attempt to fight jetlag. | 01:08 |
| anteaya | mattoliverau: you and Josh are coming to Germany? awesome!! | 01:09 |
| clarkb | anteaya: well canada hasn't qulaified in forever | 01:09 |
| clarkb | anteaya: whereas germany is fielding a good team this year | 01:09 |
| anteaya | oh no, not canada's team | 01:09 |
| mattoliverau | We've RSVP'ed... and the flights are still being organised, but should be :) | 01:09 |
| anteaya | yeah, I hope they make it to the final | 01:09 |
| anteaya | would be awesome to be in germany for that | 01:09 |
| clarkb | anteaya: thats what causes the rioting | 01:09 |
| clarkb | :) | 01:10 |
| clarkb | anteaya: you all do that with hockey | 01:10 |
| clarkb | win or lose: riots | 01:10 |
| anteaya | interesting | 01:10 |
| anteaya | I remember the vancouver riot | 01:10 |
| clarkb | mattoliverau: I am on an overnight flight and arrive ~9am Sunday | 01:10 |
| *** yaguang has joined #openstack-infra | 01:10 | |
| anteaya | and you have perceptions that rioting happens all the time | 01:10 |
| mattoliverau | clarkb: so you can join us for breakfast then :) | 01:11 |
| clarkb | mattoliverau: sure. do you plan on staying in darmstadt or frankfurt? I will be at the maritim konferenz hotel in darmstadt | 01:11 |
| *** Ryan_Lane has quit IRC | 01:12 | |
| *** aysyd has quit IRC | 01:12 | |
| clarkb | and the game should be on that night | 01:12 |
| mattoliverau | clarkb: Rackspace doesn't move too fast... So I don't know yet :) | 01:12 |
| *** camunoz has joined #openstack-infra | 01:15 | |
| *** MarkAtwood has joined #openstack-infra | 01:16 | |
| *** MarkAtwood has quit IRC | 01:16 | |
| mattoliverau | clarkb: Josh sent the wiki page to the people organising our trip, so we should be in Darmstadt somewhere :) | 01:16 |
| clarkb | cool | 01:16 |
| anteaya | wooooooo, looks good so far for the good side | 01:17 |
| openstackgerrit | Joe Gordon proposed a change to openstack/requirements: Bump minimum hacking version to 0.9.2 https://review.openstack.org/99815 | 01:18 |
| *** nosnos has joined #openstack-infra | 01:21 | |
| openstackgerrit | MORITA Kazutaka proposed a change to openstack-infra/config: Add pylint job for swift3 https://review.openstack.org/99816 | 01:22 |
| *** sarob_ has quit IRC | 01:24 | |
| *** arnaud__ has quit IRC | 01:26 | |
| *** alexpilotti has quit IRC | 01:26 | |
| *** HenryG_ has joined #openstack-infra | 01:27 | |
| *** HenryG has quit IRC | 01:29 | |
| *** markmcclain has quit IRC | 01:34 | |
| *** asettle has quit IRC | 01:37 | |
| *** rwsu has quit IRC | 01:38 | |
| *** timrc is now known as timrc-afk | 01:40 | |
| *** marcoemorais has quit IRC | 01:49 | |
| *** timrc-afk is now known as timrc | 01:50 | |
| *** thomasbiege1 has joined #openstack-infra | 01:55 | |
| *** dkehnx has quit IRC | 02:00 | |
| *** mrodden1 has quit IRC | 02:01 | |
| *** dkehnx has joined #openstack-infra | 02:01 | |
| *** marun has quit IRC | 02:01 | |
| *** marun has joined #openstack-infra | 02:02 | |
| *** mrodden has joined #openstack-infra | 02:02 | |
| *** cp16net has joined #openstack-infra | 02:09 | |
| *** zns has quit IRC | 02:11 | |
| *** zns has joined #openstack-infra | 02:24 | |
| *** dims_ has quit IRC | 02:24 | |
| *** sarob_ has joined #openstack-infra | 02:25 | |
| mestery | jogo: Noticed the ping earlier, will read the logs, if it's something urgent, ping me back. | 02:28 |
| *** sarob_ has quit IRC | 02:29 | |
| *** asettle has joined #openstack-infra | 02:33 | |
| openstackgerrit | Nikhil Manchanda proposed a change to openstack-infra/config: Added new experimental job for trove functional tests https://review.openstack.org/98517 | 02:37 |
| openstackgerrit | Nikhil Manchanda proposed a change to openstack-infra/config: Use job-template for gate-trove-buildimage jobs https://review.openstack.org/99680 | 02:37 |
| *** HenryG_ has quit IRC | 02:37 | |
| *** HenryG_ has joined #openstack-infra | 02:38 | |
| *** amcrn has quit IRC | 02:38 | |
| *** homeless has quit IRC | 02:39 | |
| *** zns has quit IRC | 02:39 | |
| *** dims_ has joined #openstack-infra | 02:39 | |
| *** malini1 has quit IRC | 02:40 | |
| *** malini1 has joined #openstack-infra | 02:40 | |
| *** zns has joined #openstack-infra | 02:41 | |
| *** tchaypo is now known as tchorizo | 02:43 | |
| *** dims_ has quit IRC | 02:44 | |
| *** zhiyan_ is now known as zhiyan | 02:45 | |
| *** arnaud has joined #openstack-infra | 02:45 | |
| *** sweston has joined #openstack-infra | 02:46 | |
| *** malini1 has quit IRC | 02:47 | |
| *** harlowja is now known as harlowja_away | 02:53 | |
| *** gargola has quit IRC | 02:54 | |
| *** amcrn has joined #openstack-infra | 02:55 | |
| *** Alexandra_ has joined #openstack-infra | 03:03 | |
| *** asettle has quit IRC | 03:04 | |
| *** plars has quit IRC | 03:05 | |
| *** gokrokve_ has joined #openstack-infra | 03:05 | |
| *** praneshp has quit IRC | 03:05 | |
| *** dims_ has joined #openstack-infra | 03:10 | |
| *** dims_ has quit IRC | 03:15 | |
| *** nosnos has quit IRC | 03:16 | |
| *** mriedem has quit IRC | 03:18 | |
| *** gyee has quit IRC | 03:19 | |
| *** gokrokve_ has quit IRC | 03:20 | |
| jogo | mestery: its not, was just amused that one of the reasons why the gate is hurting is neutron at rax | 03:23 |
| *** zns has quit IRC | 03:23 | |
| *** yamahata is now known as tacker-owner | 03:24 | |
| *** arnaud has quit IRC | 03:26 | |
| *** yjiang has joined #openstack-infra | 03:29 | |
| *** HenryG_ has quit IRC | 03:32 | |
| *** arnaud has joined #openstack-infra | 03:34 | |
| *** tacker-owner is now known as tacker-owner_ | 03:37 | |
| *** tacker-owner_ is now known as tacker-owner__ | 03:38 | |
| *** talluri has joined #openstack-infra | 03:41 | |
| *** markwash has joined #openstack-infra | 03:41 | |
| *** markwash_ has joined #openstack-infra | 03:44 | |
| *** tacker-owner__ is now known as tacker-owner | 03:45 | |
| *** markwash has quit IRC | 03:46 | |
| *** markwash_ is now known as markwash | 03:46 | |
| *** nosnos has joined #openstack-infra | 03:52 | |
| *** CaptTofu_ has quit IRC | 03:53 | |
| *** CaptTofu_ has joined #openstack-infra | 03:54 | |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/devstack-gate: Add an option to enable the nova v3 api tests https://review.openstack.org/99833 | 03:56 |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/config: Add tempest jobs with nova-v3 enabled https://review.openstack.org/99835 | 03:56 |
| *** CaptTofu_ has quit IRC | 03:58 | |
| *** tacker-owner has quit IRC | 03:59 | |
| *** thuc_ has joined #openstack-infra | 04:01 | |
| *** yamahata has joined #openstack-infra | 04:01 | |
| *** yamahata is now known as tacker-owner | 04:02 | |
| *** matsuhas_ has joined #openstack-infra | 04:03 | |
| *** markwash has quit IRC | 04:03 | |
| *** matsuhas_ has quit IRC | 04:04 | |
| *** matsuhas_ has joined #openstack-infra | 04:05 | |
| *** vladan has quit IRC | 04:05 | |
| *** matsuhashi has quit IRC | 04:05 | |
| *** alkari has joined #openstack-infra | 04:05 | |
| *** vladan has joined #openstack-infra | 04:06 | |
| *** arnaud has quit IRC | 04:09 | |
| *** dims_ has joined #openstack-infra | 04:11 | |
| *** matsuhas_ has quit IRC | 04:12 | |
| *** matsuhashi has joined #openstack-infra | 04:13 | |
| *** dims_ has quit IRC | 04:16 | |
| *** arnaud has joined #openstack-infra | 04:18 | |
| Alex_Gaynor | How does one debug a failing test like this: http://logs.openstack.org/15/99815/1/check/check-requirements-integration-dsvm/d6aefe5/console.html | 04:25 |
| clarkb | Alex_Gaynor: http://logs.openstack.org/15/99815/1/check/check-requirements-integration-dsvm/d6aefe5/logs/devstacklog.txt.gz#_2014-06-13_03_18_57_976 | 04:26 |
| Alex_Gaynor | clarkb: thanks. /me goes in search of the approrpirate bug | 04:27 |
| Alex_Gaynor | dstufft: ^ new pip can't come soon enough | 04:27 |
| dstufft | does openstack have any sort of numbers for how many times a day they are failing from connection reset by peer | 04:28 |
| Alex_Gaynor | dstufft: it's way less, most jobs don't hit pypi | 04:28 |
| dstufft | Alex_Gaynor: is openstack even using pip 1.5 yet? I thought they were still on 1.4 | 04:28 |
| Alex_Gaynor | I think it's 1.5? Either that or we manually turn on wheels, I've seen some wheels get downloaded | 04:29 |
| *** arnaud has quit IRC | 04:30 | |
| clarkb | its 1.5 for devstack | 04:31 |
| clarkb | but 1.4 for tox | 04:31 |
| clarkb | we will be 1.5 everywhere when tox upgrade happens which will happen when they release new version | 04:32 |
| *** om has joined #openstack-infra | 04:32 | |
| dstufft | I forget how the openstack log searching stuff | 04:32 |
| dstufft | works | 04:32 |
| Alex_Gaynor | elastic search | 04:33 |
| dstufft | can someone possibly look up to see what providers/regions that connection reset by peer is occuring in ? :/ | 04:33 |
| dstufft | trying to isolate it to a particular fastly DC | 04:33 |
| dstufft | if it can be isolated | 04:33 |
| clarkb | dstufft: we are having trouble with elasticsearch right now so I turned off access | 04:33 |
| dstufft | oh | 04:33 |
| dstufft | ok | 04:33 |
| clarkb | dstufft: Alex_Gaynor if you have any insight into rax cinder volume performance you might be able to help :) | 04:34 |
| clarkb | we were told pvo was the person to talk to though | 04:34 |
| dstufft | I have zero insight into any of that | 04:34 |
| *** alkari has quit IRC | 04:34 | |
| dstufft | is cinder the block storage | 04:34 |
| Alex_Gaynor | yes | 04:34 |
| dstufft | I forget the code names | 04:34 |
| dstufft | PyPI is on Rackspace and has had some problems with the block storage in IAD too fwiw | 04:34 |
| Alex_Gaynor | pvo is way more likely to be the right person than I am :/ I don't know anything about our block storage deployment. | 04:34 |
| Alex_Gaynor | I can dig more on monday if you like, I'm off tomorrow | 04:34 |
| dstufft | clarkb: is it random large IO latency spikes? | 04:35 |
| *** mmaglana has quit IRC | 04:36 | |
| clarkb | dstufft: it seems cyclic but yes | 04:36 |
| clarkb | dstufft: basisically every 6 hours or so we go through it | 04:36 |
| clarkb | iowait skyrockets | 04:37 |
| *** thuc has joined #openstack-infra | 04:37 | |
| dstufft | sounds familarish | 04:37 |
| dstufft | Not sure if ours is every 6 hours or so | 04:37 |
| dstufft | glusterfs gets real cranky | 04:37 |
| dstufft | we made it better by killing the node completely and spinning up a new server, which seemed to have a better time at it | 04:38 |
| *** saper has quit IRC | 04:38 | |
| *** tacker-owner has quit IRC | 04:38 | |
| clarkb | ya that was brought up | 04:38 |
| clarkb | we may be colocated with someone else doing crazy IO every 6 hours or some such | 04:38 |
| *** yamahata has joined #openstack-infra | 04:38 | |
| dstufft | afaik you don't get dedicated IOPS unless you're paying like 1k/month per VM | 04:39 |
| dstufft | something like that | 04:39 |
| *** yamahata is now known as tacker-owner | 04:39 | |
| clarkb | Alex_Gaynor: sure will let you know | 04:40 |
| dstufft | guess it's only $500/month | 04:40 |
| clarkb | hopefully we are sorted by then | 04:40 |
| dstufft | clarkb: Alex_Gaynor if you figure something out let me know too, it'd be nice to fix the gluterfucking | 04:40 |
| dstufft | on PyPI | 04:40 |
| *** thuc_ has quit IRC | 04:40 | |
| *** trinaths has joined #openstack-infra | 04:40 | |
| dstufft | clarkb: Openstack CI runs in which Rackspace DC, and which HP DC? | 04:44 |
| dstufft | if you don't mind me asking | 04:44 |
| *** dims_ has joined #openstack-infra | 04:46 | |
| clarkb | dstufft: rax dfw for the "static" things | 04:46 |
| clarkb | then dfw, iad, ord, for rax slaves and hpcloud east for hp slaves | 04:47 |
| Alex_Gaynor | dstufft: the last one that failed was HP | 04:47 |
| Alex_Gaynor | (reset by peer) | 04:47 |
| dstufft | ok thanks | 04:48 |
| *** talluri has quit IRC | 04:49 | |
| *** ildikov has quit IRC | 04:50 | |
| *** dims_ has quit IRC | 04:51 | |
| *** masayukig has quit IRC | 04:53 | |
| *** tacker-owner has quit IRC | 04:53 | |
| *** talluri has joined #openstack-infra | 04:55 | |
| *** jcoufal has joined #openstack-infra | 04:57 | |
| *** masayukig has joined #openstack-infra | 04:57 | |
| *** thuc has quit IRC | 04:59 | |
| *** masayukig has quit IRC | 05:02 | |
| *** SumitNaiksatam has joined #openstack-infra | 05:03 | |
| *** ramashri has joined #openstack-infra | 05:05 | |
| *** arnaud__ has joined #openstack-infra | 05:06 | |
| *** zhiyan is now known as zhiyan_ | 05:06 | |
| *** ramashri_ has joined #openstack-infra | 05:09 | |
| *** ramashri has quit IRC | 05:09 | |
| *** zhiyan_ is now known as zhiyan | 05:13 | |
| *** masayukig has joined #openstack-infra | 05:15 | |
| *** krtaylor has joined #openstack-infra | 05:24 | |
| *** ildikov has joined #openstack-infra | 05:26 | |
| *** sarob_ has joined #openstack-infra | 05:29 | |
| *** om has quit IRC | 05:29 | |
| *** sarob_ has quit IRC | 05:33 | |
| *** ihrachyshka has joined #openstack-infra | 05:35 | |
| *** ramashri_ has quit IRC | 05:38 | |
| *** talluri has quit IRC | 05:40 | |
| *** masayuki_ has quit IRC | 05:40 | |
| *** talluri has joined #openstack-infra | 05:41 | |
| *** om has joined #openstack-infra | 05:42 | |
| *** talluri has quit IRC | 05:45 | |
| *** Alexandra_ has quit IRC | 05:46 | |
| *** dims_ has joined #openstack-infra | 05:47 | |
| *** Longgeek has joined #openstack-infra | 05:47 | |
| *** e0ne has joined #openstack-infra | 05:51 | |
| *** dims_ has quit IRC | 05:51 | |
| *** rcarrill` has joined #openstack-infra | 05:51 | |
| *** matsuhashi has quit IRC | 05:52 | |
| *** rcarrillocruz has quit IRC | 05:53 | |
| *** rcarrillocruz has joined #openstack-infra | 05:54 | |
| *** sarob_ has joined #openstack-infra | 05:54 | |
| *** rcarrill` has quit IRC | 05:56 | |
| *** matsuhashi has joined #openstack-infra | 05:57 | |
| *** sarob_ has quit IRC | 05:59 | |
| *** jeremyb has quit IRC | 06:00 | |
| *** e0ne has quit IRC | 06:02 | |
| *** e0ne has joined #openstack-infra | 06:03 | |
| *** praneshp has joined #openstack-infra | 06:03 | |
| *** cody-somerville has quit IRC | 06:04 | |
| *** yamahata has joined #openstack-infra | 06:04 | |
| openstackgerrit | Ian Wienand proposed a change to openstack-infra/devstack-gate: [WIP] cleanup of log copy https://review.openstack.org/99852 | 06:05 |
| *** arnaud__ has quit IRC | 06:06 | |
| *** talluri has joined #openstack-infra | 06:06 | |
| *** e0ne has quit IRC | 06:08 | |
| *** basha has joined #openstack-infra | 06:09 | |
| *** praneshp_ has joined #openstack-infra | 06:10 | |
| *** camunoz has quit IRC | 06:11 | |
| *** praneshp has quit IRC | 06:13 | |
| *** praneshp_ is now known as praneshp | 06:13 | |
| *** cody-somerville has joined #openstack-infra | 06:15 | |
| *** zehicle_at_dell has joined #openstack-infra | 06:16 | |
| *** om has quit IRC | 06:30 | |
| *** doude has joined #openstack-infra | 06:30 | |
| *** ihrachyshka has quit IRC | 06:30 | |
| *** zhiyan is now known as zhiyan_ | 06:32 | |
| *** dkehn_ has joined #openstack-infra | 06:34 | |
| *** penguinRaider has joined #openstack-infra | 06:36 | |
| *** e0ne has joined #openstack-infra | 06:38 | |
| *** dkehnx has quit IRC | 06:38 | |
| *** e0ne has quit IRC | 06:39 | |
| *** e0ne has joined #openstack-infra | 06:40 | |
| *** bogdando has joined #openstack-infra | 06:42 | |
| *** e0ne has quit IRC | 06:42 | |
| *** dims_ has joined #openstack-infra | 06:47 | |
| *** flaper87|afk is now known as flaper87 | 06:49 | |
| *** om has joined #openstack-infra | 06:51 | |
| *** dims_ has quit IRC | 06:54 | |
| *** talluri has quit IRC | 06:54 | |
| *** talluri has joined #openstack-infra | 06:55 | |
| *** cody-somerville has quit IRC | 06:57 | |
| *** talluri has quit IRC | 06:59 | |
| *** zhiyan_ is now known as zhiyan | 07:00 | |
| *** rdopieralski has joined #openstack-infra | 07:01 | |
| *** jlibosva has joined #openstack-infra | 07:03 | |
| *** srenatus has quit IRC | 07:06 | |
| *** srenatus has joined #openstack-infra | 07:06 | |
| *** doude has quit IRC | 07:07 | |
| *** doude has joined #openstack-infra | 07:07 | |
| *** basha has quit IRC | 07:08 | |
| *** mrmartin has joined #openstack-infra | 07:09 | |
| *** cody-somerville has joined #openstack-infra | 07:09 | |
| *** achuprin_ has quit IRC | 07:15 | |
| *** _nadya_ has joined #openstack-infra | 07:16 | |
| *** jamielennox is now known as jamielennox|away | 07:17 | |
| *** andreykurilin_ has joined #openstack-infra | 07:21 | |
| *** dkehn has quit IRC | 07:22 | |
| *** medieval1 has quit IRC | 07:23 | |
| *** dkehn_ has quit IRC | 07:23 | |
| *** dkehn has joined #openstack-infra | 07:23 | |
| *** mrda is now known as mrda-weekend | 07:25 | |
| *** medieval1 has joined #openstack-infra | 07:26 | |
| *** dkehn_ has joined #openstack-infra | 07:28 | |
| *** afazekas_ has joined #openstack-infra | 07:28 | |
| *** achuprin_ has joined #openstack-infra | 07:30 | |
| *** matsuhas_ has joined #openstack-infra | 07:36 | |
| *** matsuhashi has quit IRC | 07:36 | |
| *** amcrn has quit IRC | 07:37 | |
| *** andreykurilin_ has quit IRC | 07:37 | |
| *** andreykurilin_ has joined #openstack-infra | 07:38 | |
| *** ihrachyshka has joined #openstack-infra | 07:38 | |
| *** medieval1 has quit IRC | 07:39 | |
| *** dkehn has quit IRC | 07:39 | |
| *** dkehn_ has quit IRC | 07:39 | |
| *** hashar has joined #openstack-infra | 07:39 | |
| *** om has quit IRC | 07:41 | |
| *** medieval1 has joined #openstack-infra | 07:45 | |
| *** dkehn has joined #openstack-infra | 07:45 | |
| *** dkehn_ has joined #openstack-infra | 07:46 | |
| *** cody-somerville has quit IRC | 07:50 | |
| *** dims_ has joined #openstack-infra | 07:50 | |
| *** isviridov|away is now known as isviridpv | 07:55 | |
| *** isviridpv is now known as isviridov | 07:55 | |
| *** dims_ has quit IRC | 07:55 | |
| *** talluri has joined #openstack-infra | 07:55 | |
| *** sarob_ has joined #openstack-infra | 07:57 | |
| *** Hal has joined #openstack-infra | 07:59 | |
| *** Hal is now known as Guest47184 | 07:59 | |
| *** talluri has quit IRC | 08:00 | |
| *** sarob_ has quit IRC | 08:01 | |
| *** mrmartin has quit IRC | 08:03 | |
| *** cody-somerville has joined #openstack-infra | 08:04 | |
| openstackgerrit | Bob Ball proposed a change to openstack-infra/devstack-gate: Merge ZUUL_REF branch https://review.openstack.org/99863 | 08:05 |
| BobBall | hmmm - that's not where it's meant to be going - sorry | 08:07 |
| *** marios has joined #openstack-infra | 08:07 | |
| *** zhiyan is now known as zhiyan_ | 08:10 | |
| *** srenatus has quit IRC | 08:12 | |
| *** srenatus has joined #openstack-infra | 08:13 | |
| *** tkelsey has joined #openstack-infra | 08:13 | |
| *** e0ne has joined #openstack-infra | 08:16 | |
| *** dizquierdo has joined #openstack-infra | 08:16 | |
| *** e0ne_ has joined #openstack-infra | 08:16 | |
| *** derekh_ has joined #openstack-infra | 08:17 | |
| *** amcrn has joined #openstack-infra | 08:18 | |
| *** e0ne has quit IRC | 08:20 | |
| *** trinaths has quit IRC | 08:23 | |
| *** pblaho has joined #openstack-infra | 08:26 | |
| *** e0ne_ has quit IRC | 08:28 | |
| *** e0ne has joined #openstack-infra | 08:28 | |
| derekh_ | SergeyLukjanov: I'm not seeing anything going through the check-tripleo queue, would you be able to see if you can see any problemns your end? | 08:28 |
| derekh_ | hmm there is a template that started building 2 dates ago# | 08:30 |
| *** zhiyan_ is now known as zhiyan | 08:30 | |
| derekh_ | but nodepool seems to have 17 instances, one in ERROR state | 08:31 |
| *** penguinRaider has quit IRC | 08:34 | |
| *** cody-somerville has quit IRC | 08:39 | |
| *** eglynn-office has joined #openstack-infra | 08:41 | |
| eglynn-office | good morning folks! | 08:42 |
| eglynn-office | FYI logstash appears to be down ... http://logstash.openstack.org/ | 08:43 |
| *** pelix has joined #openstack-infra | 08:44 | |
| derekh_ | tcpdump on tripleo rack show no traffic at all the API, as if nodepool isn't trying | 08:46 |
| openstackgerrit | Thierry Carrez proposed a change to openstack-infra/release-tools: Add script for new-style milestone publication https://review.openstack.org/98123 | 08:49 |
| ttx | SergeyLukjanov: I updated the script -- this is the version I ended up using for juno-1 ^ | 08:50 |
| *** talluri has joined #openstack-infra | 08:50 | |
| ttx | SergeyLukjanov: I'd love your +1 on it before I self-approve it | 08:50 |
| *** dims_ has joined #openstack-infra | 08:51 | |
| *** cody-somerville has joined #openstack-infra | 08:51 | |
| *** zhiyan is now known as zhiyan_ | 08:52 | |
| *** rcarrill` has joined #openstack-infra | 08:54 | |
| *** johnthetubaguy has quit IRC | 08:55 | |
| *** thomasbiege1 has left #openstack-infra | 08:55 | |
| *** rcarrillocruz has quit IRC | 08:55 | |
| *** dims_ has quit IRC | 08:56 | |
| *** johnthetubaguy has joined #openstack-infra | 08:56 | |
| *** johnthetubaguy has quit IRC | 08:57 | |
| *** johnthetubaguy has joined #openstack-infra | 08:57 | |
| *** johnthetubaguy has quit IRC | 08:57 | |
| *** matsuhas_ has quit IRC | 08:57 | |
| *** zhiyan_ is now known as zhiyan | 09:00 | |
| *** johnthetubaguy has joined #openstack-infra | 09:02 | |
| *** johnthetubaguy has quit IRC | 09:03 | |
| *** rdopieralski is now known as rdopiera | 09:05 | |
| *** johnthetubaguy has joined #openstack-infra | 09:05 | |
| *** basha has joined #openstack-infra | 09:06 | |
| *** rcarrill` is now known as rcarrillocruz | 09:06 | |
| *** CaptTofu_ has joined #openstack-infra | 09:07 | |
| *** zhiyan is now known as zhiyan_ | 09:09 | |
| *** d0ugal has quit IRC | 09:09 | |
| *** d0ugal has joined #openstack-infra | 09:10 | |
| *** alkari has joined #openstack-infra | 09:10 | |
| *** zhiyan_ is now known as zhiyan | 09:10 | |
| *** matsuhashi has joined #openstack-infra | 09:12 | |
| *** CaptTofu_ has quit IRC | 09:12 | |
| *** talluri has quit IRC | 09:13 | |
| isviridov | ttx, as far as I know it is public holyday in Russia till Monday | 09:13 |
| *** talluri has joined #openstack-infra | 09:13 | |
| isviridov | ttx, so SergeyLukjanov is not available. Also need him | 09:14 |
| ttx | hah, no pb | 09:14 |
| *** alkari has quit IRC | 09:15 | |
| isviridov | ttx, BTW do infra team creating new projects only every Friday? Is today one of them? | 09:17 |
| *** ildikov has quit IRC | 09:17 | |
| *** ildikov has joined #openstack-infra | 09:17 | |
| *** talluri has quit IRC | 09:18 | |
| ttx | isviridov: they usually do it over weekends yes. Not sure this is one of them | 09:18 |
| * isviridov will ask in the evening | 09:20 | |
| *** ominakov has joined #openstack-infra | 09:21 | |
| *** habib has joined #openstack-infra | 09:24 | |
| mattoliverau | Phew, a long day, I'm calling it a night, have a great weekend all. | 09:25 |
| *** mkerrin has quit IRC | 09:26 | |
| *** johnthetubaguy_ has joined #openstack-infra | 09:30 | |
| *** johnthetubaguy has quit IRC | 09:31 | |
| *** johnthetubaguy_ is now known as johnthetubaguy | 09:31 | |
| *** mkerrin has joined #openstack-infra | 09:33 | |
| *** Guest47184 has quit IRC | 09:38 | |
| *** Guest47184 has joined #openstack-infra | 09:41 | |
| derekh_ | Anybody have any thoughts on why nothing is running in the check-tripleo queue | 09:41 |
| *** praneshp has quit IRC | 09:42 | |
| *** matsuhashi has quit IRC | 09:43 | |
| *** talluri has joined #openstack-infra | 09:44 | |
| openstackgerrit | A change was merged to openstack-infra/config: Be specific about which ES nodes are puppetable https://review.openstack.org/99794 | 09:45 |
| *** talluri_ has joined #openstack-infra | 09:45 | |
| *** maxbit has quit IRC | 09:46 | |
| *** talluri has quit IRC | 09:48 | |
| *** yamahata_ has quit IRC | 09:49 | |
| gilliard | Hi all - is there something wrong with nodepool? Looks like >50% instances are "deleting" | 09:50 |
| *** talluri_ has quit IRC | 09:50 | |
| *** dims_ has joined #openstack-infra | 09:52 | |
| *** dims_ has quit IRC | 09:56 | |
| *** ccorrigan has quit IRC | 09:57 | |
| *** sarob_ has joined #openstack-infra | 09:58 | |
| *** talluri has joined #openstack-infra | 09:59 | |
| derekh_ | gilliard: somethings seems up alright, its not using the tripleo region at all | 10:02 |
| *** sarob_ has quit IRC | 10:02 | |
| openstackgerrit | A change was merged to openstack-infra/config: Remove tripleo cross-testing with oslotest https://review.openstack.org/92910 | 10:02 |
| *** dkehn__ has joined #openstack-infra | 10:04 | |
| *** rlandy has joined #openstack-infra | 10:04 | |
| *** om has joined #openstack-infra | 10:04 | |
| *** jamielennox|away has quit IRC | 10:04 | |
| *** zhiyan is now known as zhiyan_ | 10:06 | |
| gilliard | I haven't any way to investigate nodepool itself but if anyone suspects problems with HP instances just give me a shout :) | 10:06 |
| *** jamielennox|away has joined #openstack-infra | 10:07 | |
| openstackgerrit | Thierry Carrez proposed a change to openstack-infra/release-tools: Support swift and oslo milestone releases https://review.openstack.org/99892 | 10:08 |
| *** dkehn has quit IRC | 10:08 | |
| *** AaronGr has quit IRC | 10:09 | |
| *** AaronGr has joined #openstack-infra | 10:11 | |
| *** yamahata has quit IRC | 10:18 | |
| *** e0ne has quit IRC | 10:22 | |
| *** e0ne has joined #openstack-infra | 10:23 | |
| *** Longgeek has quit IRC | 10:26 | |
| *** nosnos has quit IRC | 10:26 | |
| *** Longgeek has joined #openstack-infra | 10:26 | |
| *** e0ne has quit IRC | 10:27 | |
| *** hashar has quit IRC | 10:27 | |
| *** Longgeek_ has joined #openstack-infra | 10:27 | |
| sdague | I guess we never turned the api back on for elastic search | 10:28 |
| *** Longgeek has quit IRC | 10:28 | |
| *** e0ne has joined #openstack-infra | 10:28 | |
| sdague | gilliard: I think the issue is more on the rax side right now | 10:28 |
| gilliard | sdague: OK. Do you have a graphite link that splits up the nodepool data by cloud? | 10:29 |
| sdague | I don't | 10:30 |
| * gilliard goes to poke around in graphite. | 10:30 | |
| sdague | honestly, I just ask the infra folks about nodepool when I think it's bonkers :) | 10:30 |
| sdague | maybe SergeyLukjanov would know | 10:30 |
| gilliard | No probs. I think it'd be useful to have some info about nodepool on our wall somewhere | 10:31 |
| *** markmc has joined #openstack-infra | 10:33 | |
| *** Guest47184 has quit IRC | 10:33 | |
| *** nosnos has joined #openstack-infra | 10:34 | |
| *** ociuhandu has joined #openstack-infra | 10:35 | |
| derekh_ | sdague: If you have a chance would you mind looking to see why nodepool doesn't seem to be using the tripleo region | 10:37 |
| derekh_ | sdague: or is it todo with the RAX problem you mentioned | 10:37 |
| openstackgerrit | A change was merged to openstack-infra/config: Remove link to devstack_launch_slave.pp https://review.openstack.org/97110 | 10:38 |
| sdague | derekh_: I don't have access to nodepool | 10:38 |
| lifeless | derekh_: you want SergeyLukjanov | 10:38 |
| derekh_ | sdague: ahh ok, np | 10:38 |
| sdague | it can't be related to the rax issue | 10:38 |
| *** zehicle_at_dell has quit IRC | 10:38 | |
| lifeless | well, it could, but very very unlikely given nodepools internals | 10:38 |
| derekh_ | lifeless: I pinged SergeyLukjanov earlier he doesn't seem to be around | 10:38 |
| sdague | that's just the fact that rax has a ton of nodes in 'deleting' state that are stuck | 10:38 |
| *** e0ne has quit IRC | 10:38 | |
| *** nosnos has quit IRC | 10:39 | |
| sdague | and we keep looping and retrying them | 10:39 |
| sdague | but it doesn't matter, because you can't delete a deleting node | 10:39 |
| lifeless | I wonder if we hit the socket timeout wedge again | 10:39 |
| lifeless | on the rh1 region this time | 10:39 |
| *** e0ne has joined #openstack-infra | 10:39 | |
| lifeless | sdague: you will be able to once jogos patch lands | 10:40 |
| derekh_ | lifeless: possibly, we havn't seen it in that region befor I think but that doesn't mean much | 10:40 |
| sdague | lifeless: and once it gets into the clouds :) | 10:40 |
| lifeless | sdague: ponies! | 10:40 |
| lifeless | sdague: what are you doing up? Am I up too late? | 10:40 |
| sdague | I'm usually up this early :) | 10:41 |
| sdague | I tend to start my day at 6am EST | 10:41 |
| *** Alexei_987 has quit IRC | 10:41 | |
| *** denis_makogon has joined #openstack-infra | 10:45 | |
| openstackgerrit | Lukas Bednar proposed a change to openstack-infra/jenkins-job-builder: multijob: added kill-phase-on option https://review.openstack.org/99903 | 10:45 |
| *** _nadya_ has quit IRC | 10:45 | |
| gilliard | Thu/Fri are public holidays in Russia, SergeyLukjanov is out till Mon I think. | 10:46 |
| *** chandan_kumar has joined #openstack-infra | 10:51 | |
| *** chandankumar has quit IRC | 10:52 | |
| sdague | gilliard: gotcha | 10:52 |
| *** radez_g0n3 is now known as radez | 10:52 | |
| *** dims_ has joined #openstack-infra | 10:52 | |
| *** chandan_kumar has quit IRC | 10:56 | |
| *** chandan_kumar has joined #openstack-infra | 10:57 | |
| *** dims_ has quit IRC | 10:57 | |
| *** _nadya_ has joined #openstack-infra | 11:03 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add a mailing list for Win The Enterprise WG https://review.openstack.org/98762 | 11:06 |
| *** julim has joined #openstack-infra | 11:08 | |
| *** yjiang has quit IRC | 11:11 | |
| *** trinaths has joined #openstack-infra | 11:12 | |
| *** ildikov_ has joined #openstack-infra | 11:13 | |
| *** ildikov has quit IRC | 11:14 | |
| *** dims_ has joined #openstack-infra | 11:16 | |
| *** _nadya_ has quit IRC | 11:19 | |
| trinaths | anteaya: hello | 11:20 |
| enikanorov | whatsup with logstash? "If it helps, I received a 0 error from: " | 11:22 |
| *** marios has left #openstack-infra | 11:23 | |
| *** viktors|afk is now known as viktors | 11:26 | |
| *** tkelsey has quit IRC | 11:27 | |
| trinaths | Got a question on gerrit change sets. how to know by command line that a specific change is merged ? any ideas | 11:28 |
| *** hashar has joined #openstack-infra | 11:29 | |
| *** thomasbiege has joined #openstack-infra | 11:31 | |
| *** thomasbiege has quit IRC | 11:32 | |
| *** talluri has quit IRC | 11:33 | |
| sdague | is there anyone with gerrit admin here? | 11:33 |
| sdague | anteaya: do you have that yet? | 11:33 |
| *** talluri has joined #openstack-infra | 11:33 | |
| sdague | I need to get added to - https://review.openstack.org/#/admin/groups/363,members when there is a chance | 11:34 |
| *** srenatus has quit IRC | 11:35 | |
| *** srenatus has joined #openstack-infra | 11:35 | |
| *** zul has quit IRC | 11:36 | |
| *** talluri has quit IRC | 11:38 | |
| *** thomasbiege has joined #openstack-infra | 11:42 | |
| openstackgerrit | Thierry Carrez proposed a change to openstack-infra/infra-specs: Added specification for storyboard story tags https://review.openstack.org/97211 | 11:44 |
| *** yamahata has joined #openstack-infra | 11:46 | |
| openstackgerrit | Thierry Carrez proposed a change to openstack-infra/infra-specs: Added specification for storyboard story tags https://review.openstack.org/97211 | 11:46 |
| openstackgerrit | Thierry Carrez proposed a change to openstack-infra/infra-specs: Added specification for storyboard story tags https://review.openstack.org/97211 | 11:47 |
| ttx | sorry for the noise | 11:47 |
| *** yamahata is now known as tacker-owner | 11:47 | |
| *** matjazp has joined #openstack-infra | 11:48 | |
| *** tacker-owner is now known as yamahata | 11:49 | |
| *** _nadya_ has joined #openstack-infra | 11:49 | |
| *** trinaths has quit IRC | 11:50 | |
| *** yamahata is now known as tacker-owner | 11:53 | |
| *** trinaths has joined #openstack-infra | 11:54 | |
| *** basha_ has joined #openstack-infra | 11:55 | |
| *** tacker-owner has quit IRC | 11:55 | |
| trinaths | fungi: hi | 11:55 |
| *** yamahata has joined #openstack-infra | 11:56 | |
| *** basha has quit IRC | 11:56 | |
| *** basha_ is now known as basha | 11:56 | |
| *** dizquierdo has quit IRC | 11:56 | |
| *** maxbit has joined #openstack-infra | 11:56 | |
| *** lcostantino has joined #openstack-infra | 11:57 | |
| *** weshay has joined #openstack-infra | 12:01 | |
| fungi | sdague: adding you now | 12:01 |
| trinaths | fungi: hi | 12:02 |
| trinaths | fungi: need some help on jenkins job builder. | 12:02 |
| fungi | sdague: done | 12:02 |
| *** mbacchi has joined #openstack-infra | 12:02 | |
| fungi | trinaths: what's the trouble? | 12:03 |
| *** mwagner_lap has quit IRC | 12:03 | |
| trinaths | fungi: i'm using single jenkins node for devstack. now I have another node in place and its sync with jenkins master. | 12:04 |
| trinaths | fungi: and now when there are serveral jobs in queue from zuul, I need to run 2 jobs simultaneously. | 12:04 |
| trinaths | fungi: can you help me on how can I achieve this.? guidelines please | 12:05 |
| trinaths | fungi: my jenkins_jobs/config/projects.yaml looks like, http://paste.openstack.org/show/83934/ | 12:06 |
| *** ArxCruz has joined #openstack-infra | 12:06 | |
| fungi | trinaths: you would either need to assign some jobs to separate slave node names, or set a common slave label on them | 12:07 |
| fungi | trinaths: for openstack infra, we use common sets of platform-specific node labels to accomplish that | 12:08 |
| trinaths | fungi: okay. my old slave-node label is 'jenkins_slave'. for the new one I named 'jenkins_slave_2'. If I move the new one's lable to 'jenkins_slave', I can run two jobs simultaneously. right ? | 12:09 |
| fungi | trinaths: yes, or i think you can use a boolean operator on the node line to specify multiple names | 12:10 |
| fungi | if memory serves it would be like node: jenkins_slave || jenkins_slave_2 | 12:11 |
| trinaths | fungi: okay | 12:11 |
| *** dprince has joined #openstack-infra | 12:11 | |
| trinaths | fungi: thanks for the help | 12:11 |
| *** yaguang has quit IRC | 12:12 | |
| *** basha has quit IRC | 12:12 | |
| sdague | fungi: can you also turn apache back on for logstash.openstack.org? | 12:13 |
| fungi | trinaths: yeah, while we do use labels, we sometimes use the || operator to specify more than one label, for example http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/jenkins_job_builder/config/projects.yaml#n309 | 12:13 |
| fungi | sdague: sure | 12:13 |
| sdague | thanks sir | 12:13 |
| *** eharney has joined #openstack-infra | 12:14 | |
| sdague | fungi: hmm... being in the ptl group doesn't put me in the core group? | 12:15 |
| sdague | I can't seem to modify this - https://review.openstack.org/#/admin/groups/362,members | 12:15 |
| sdague | or is this an issue about possibly having multiple ids again in gerrit? | 12:15 |
| fungi | sdague: no, they're just separate groups | 12:16 |
| *** ihrachyshka has quit IRC | 12:16 | |
| fungi | i added you to that one too now | 12:16 |
| *** ihrachyshka has joined #openstack-infra | 12:16 | |
| fungi | you can make one included in the other or something if it suits you | 12:16 |
| *** bookwar has joined #openstack-infra | 12:17 | |
| *** trinaths has quit IRC | 12:17 | |
| *** CaptTofu_ has joined #openstack-infra | 12:17 | |
| fungi | sdague: and kibana should be reachable again now | 12:17 |
| *** om has quit IRC | 12:19 | |
| *** aysyd has joined #openstack-infra | 12:20 | |
| *** rfolco has joined #openstack-infra | 12:20 | |
| *** mriedem has joined #openstack-infra | 12:22 | |
| *** talluri has joined #openstack-infra | 12:23 | |
| sdague | fungi: thanks | 12:24 |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Fix ntp restart for Fedora https://review.openstack.org/98288 | 12:25 |
| *** mfer has joined #openstack-infra | 12:26 | |
| *** yamahata has quit IRC | 12:28 | |
| *** alexpilotti has joined #openstack-infra | 12:30 | |
| *** talluri has quit IRC | 12:31 | |
| *** talluri has joined #openstack-infra | 12:32 | |
| *** radez is now known as radez_g0n3 | 12:33 | |
| *** talluri has quit IRC | 12:37 | |
| *** tkelsey has joined #openstack-infra | 12:38 | |
| *** om has joined #openstack-infra | 12:39 | |
| openstackgerrit | Peter Belanyi proposed a change to openstack-infra/config: Add jshint job for tuskar-ui https://review.openstack.org/96473 | 12:42 |
| *** Guest47184 has joined #openstack-infra | 12:44 | |
| *** CaptTofu_ has quit IRC | 12:48 | |
| *** ildikov has joined #openstack-infra | 12:52 | |
| *** ildikov_ has quit IRC | 12:52 | |
| derekh_ | fungi: if you have a few minutes could you see why nothing seems to be running in the check-tripleo queue | 12:56 |
| *** matjazp has quit IRC | 12:59 | |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/config: Don't run large-ops test on stable/havana branches https://review.openstack.org/99750 | 13:00 |
| *** med_ has joined #openstack-infra | 13:01 | |
| *** med_ has quit IRC | 13:01 | |
| *** med_ has joined #openstack-infra | 13:01 | |
| *** adalbas has joined #openstack-infra | 13:03 | |
| *** om has quit IRC | 13:05 | |
| *** HenryG has joined #openstack-infra | 13:07 | |
| *** russellb is now known as rustlebee | 13:07 | |
| openstackgerrit | Valeriy Ponomaryov proposed a change to openstack-infra/config: Added devstack job for manila https://review.openstack.org/99933 | 13:11 |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/config: Add tempest jobs with nova-v3 enabled https://review.openstack.org/99835 | 13:17 |
| *** hashar has quit IRC | 13:17 | |
| *** yolanda has quit IRC | 13:22 | |
| *** smarcet has joined #openstack-infra | 13:23 | |
| *** blamar has quit IRC | 13:26 | |
| *** matjazp has joined #openstack-infra | 13:26 | |
| *** yolanda has joined #openstack-infra | 13:26 | |
| *** radez_g0n3 is now known as radez | 13:26 | |
| *** lbragstad has joined #openstack-infra | 13:26 | |
| *** basha has joined #openstack-infra | 13:27 | |
| *** matjazp has quit IRC | 13:27 | |
| *** reaper has quit IRC | 13:28 | |
| *** CaptTofu_ has joined #openstack-infra | 13:28 | |
| sdague | fungi: I also think a pretty substantial actor in our failures is rax networking | 13:30 |
| *** lbragstad has quit IRC | 13:31 | |
| sdague | the network timeouts talking to even our own pip mirror are pretty high | 13:31 |
| *** homeless has joined #openstack-infra | 13:31 | |
| *** rdopiera has quit IRC | 13:31 | |
| *** dkehn__ is now known as dkehnx | 13:31 | |
| *** zul has joined #openstack-infra | 13:32 | |
| *** dkehn_ is now known as dkehn | 13:32 | |
| *** basha has quit IRC | 13:32 | |
| fungi | derekh_: i'll try, but i'm in the middle of dismantling all my servers and network, so i'm about to go dark for a bit of the day | 13:33 |
| derekh_ | fungi: ok, cool | 13:34 |
| *** rdopieralski has joined #openstack-infra | 13:36 | |
| *** vhoward has joined #openstack-infra | 13:37 | |
| *** dkliban_afk is now known as dkliban | 13:39 | |
| StevenK | fungi: Moving, or something more sinister? | 13:41 |
| *** malini1 has joined #openstack-infra | 13:42 | |
| *** malini1 has left #openstack-infra | 13:43 | |
| fungi | StevenK: sinister moving | 13:44 |
| StevenK | fungi: Haha | 13:44 |
| johnthetubaguy | sdague: the issues with the mirrors came up before, do you guys have the info you need to make support talk to you, to try fix that? | 13:46 |
| sdague | johnthetubaguy: honestly, I have no idea | 13:46 |
| johnthetubaguy | sdague: is it talking to the ubuntu mirror, or the rackspace mirror? | 13:46 |
| sdague | johnthetubaguy: both, as well as pypi.openstack.org | 13:47 |
| sdague | which is hosted in rax | 13:47 |
| johnthetubaguy | I think you have to change the fqdn of the mirror to reach hours, its not like a transaprent procsy | 13:47 |
| johnthetubaguy | hmm, that sucks | 13:47 |
| johnthetubaguy | between DC will just be regular internet traffic I think | 13:47 |
| johnthetubaguy | so you might want one per DC | 13:47 |
| johnthetubaguy | and put it onto service net, not the public network | 13:47 |
| sdague | well, that doesn't help the hp regions | 13:47 |
| johnthetubaguy | but agreed, you shouldn't see those problems anyway | 13:48 |
| johnthetubaguy | right | 13:48 |
| johnthetubaguy | sdague: how big is your server? it might be our QoS on your networking throttling things | 13:48 |
| sdague | johnthetubaguy: I know none of these details actually | 13:48 |
| fungi | sdague: johnthetubaguy: yeah, we've talked about quite possibly putting mirrors in each region of each provider we use | 13:48 |
| sdague | I can just see them from the fail logs | 13:48 |
| johnthetubaguy | sdague: true true | 13:49 |
| fungi | i think we agree it's probably a good way to mitigate it, just need someone with available time to get it all puppeted | 13:49 |
| johnthetubaguy | fungi: how about resize up to 30Gb instance | 13:49 |
| johnthetubaguy | that way you get the full box of network (in standard thats 1Gb link) | 13:49 |
| johnthetubaguy | should stop any OVS issues from noisy friends, and reduce the risk of you hitting QoS limits | 13:50 |
| sdague | johnthetubaguy: the issue is, right now, we've got 112 race bugs we are tracking in elastic recheck | 13:50 |
| fungi | johnthetubaguy: that's not a bad idea actually | 13:50 |
| sdague | and seem to be adding 3 or 4 a day | 13:50 |
| *** hashar has joined #openstack-infra | 13:50 | |
| sdague | so diving really deep on any particular one is hard in the firehose :( | 13:50 |
| johnthetubaguy | sdague: understood | 13:51 |
| *** jistr has joined #openstack-infra | 13:51 | |
| *** _nadya_ has quit IRC | 13:51 | |
| johnthetubaguy | fungi: I would resize up to 30Gb standard, or snapshot and rebuild, I guess, to stop the outage, then see if that helps | 13:51 |
| johnthetubaguy | fungi: its a slight stab in the dark, I know, but should be a quick one | 13:52 |
| *** mwagner_lap has joined #openstack-infra | 13:52 | |
| srenatus | hmm having problems up update a `contact information` on gerrit right now... known issue? | 13:53 |
| *** matjazp has joined #openstack-infra | 13:54 | |
| fungi | srenatus: the error message got more vague when we upgraded gerrit recently, but make sure you've followed the suggestions at https://wiki.openstack.org/wiki/CLA-FAQ#When_trying_to_sign_the_new_ICLA_and_include_contact_information.2C_why_am_I.27m_getting_an_error_message_saying_that_my_E-mail_address_doesn.27t_correspond_to_a_Foundation_membership.3F | 13:54 |
| fungi | srenatus: ideally you were following the instructions at https://wiki.openstack.org/wiki/How_To_Contribute#Contributor_License_Agreement which would tell you to sign up for a free openstack foundation membership before moving onto gerrit | 13:56 |
| srenatus | fungi: thanks for the pointer, checking that | 13:56 |
| fungi | srenatus: the primary/preferred e-mail address you listed in your foundation profile and your gerrit settings need to match initially while you're submitting contact into in gerrit, to link the accounts | 13:57 |
| *** _nadya_ has joined #openstack-infra | 13:57 | |
| *** annegentlereally has joined #openstack-infra | 13:57 | |
| fungi | srenatus: we've got efforts underway to simplify all this, so apologies for the complication there | 13:57 |
| srenatus | :) no problems | 13:57 |
| srenatus | everything works fine for me, anyways, just trying to get a coworker set up wit gerrit... | 13:58 |
| srenatus | couldn't remember all the steps. | 13:58 |
| srenatus | thanks again | 13:58 |
| fungi | srenatus: you're welcome! | 13:58 |
| *** zz_gondoi is now known as gondoi | 13:59 | |
| *** yamahata has joined #openstack-infra | 14:00 | |
| *** yamahata has quit IRC | 14:01 | |
| fungi | derekh_: the only 18 nodes i see are in the red hat region, and they're all in a delete state, most for roughly a day at this point | 14:01 |
| *** yamahata has joined #openstack-infra | 14:01 | |
| *** basha has joined #openstack-infra | 14:02 | |
| *** dizquierdo has joined #openstack-infra | 14:02 | |
| *** rdopieralski has quit IRC | 14:03 | |
| derekh_ | fungi: ya, we arn't using the other region at the moment, I see all of those nodes (except 1 which is error) in a running state | 14:03 |
| *** thomasbiege has left #openstack-infra | 14:03 | |
| fungi | ConnectionError: HTTPSConnectionPool(host='ci-overcloud.rh1.tripleo.org', port=13000): Max retries exceeded with url: /v2.0/tokens (Caused by <class 'socket.error'>: [Errno 110] Connection timed out) | 14:03 |
| fungi | derekh_: that looks like the probable culprit | 14:03 |
| derekh_ | fungi: is that currently happening or was it back then it tried to delete them? | 14:04 |
| fungi | and now i have to disappear for a while... unplugging my networks here and whatnot. should be back in a couple hours, hopefully | 14:04 |
| fungi | derekh_: that was at 2014-06-13 07:23:55,014 utc | 14:04 |
| fungi | one of the delete attempts | 14:04 |
| fungi | most recent mention of tripleo in the nodepool log i could find | 14:05 |
| * fungi vanishes in a puff of NO CARRIER | 14:05 | |
| derekh_ | fungi: ok, I'm not seeing any api traffic at all from nodepool | 14:05 |
| derekh_ | fungi: thanks, I'll see if I can grab somebody else when their on | 14:05 |
| *** jistr has quit IRC | 14:06 | |
| *** jistr has joined #openstack-infra | 14:07 | |
| *** matjazp has quit IRC | 14:08 | |
| *** jcoufal has quit IRC | 14:08 | |
| *** blamar has joined #openstack-infra | 14:11 | |
| srenatus | just learnt that you can do interesting things by ssh'ing into review.openstack.org and using `gerrit`. Is that ok in general or is there a prefered API to use for this data? | 14:13 |
| sdague | srenatus: that's the documented api | 14:13 |
| sdague | there is also now a rest api | 14:13 |
| sdague | but it's totally legit | 14:14 |
| anteaya | srenatus: you can only run the commands that your user is allowed given the permissions on that user | 14:14 |
| *** thedodd has joined #openstack-infra | 14:15 | |
| sdague | derekh_: is there an external to RH monitoring job on your endpoints? so that it can at least isolate that piece | 14:15 |
| *** matjazp has joined #openstack-infra | 14:16 | |
| *** gondoi is now known as zz_gondoi | 14:19 | |
| *** _nadya_ has quit IRC | 14:20 | |
| derekh_ | sdague: nothing monitoring it, but I can access it from outside RH | 14:21 |
| sdague | derekh_: well, if there was something that was hitting it more regularly, it might exposed the blips. The one off checks don't usually catch those. | 14:21 |
| *** CaptTofu_ has quit IRC | 14:22 | |
| sdague | especially as this seems to be a more regular occurance | 14:22 |
| *** ihrachyshka has quit IRC | 14:22 | |
| derekh_ | sdague: yup agreed, we have a whole bunch of things in the CI spec to get through, thats one of them | 14:22 |
| *** reaper has joined #openstack-infra | 14:24 | |
| *** basha has quit IRC | 14:24 | |
| *** trinaths has joined #openstack-infra | 14:25 | |
| *** matjazp has quit IRC | 14:25 | |
| *** sandywalsh_ has joined #openstack-infra | 14:27 | |
| *** [1]trinaths has joined #openstack-infra | 14:27 | |
| *** zns has joined #openstack-infra | 14:29 | |
| *** trinaths has quit IRC | 14:29 | |
| *** [1]trinaths is now known as trinaths | 14:29 | |
| *** cp16net_ has joined #openstack-infra | 14:32 | |
| *** homeless has quit IRC | 14:33 | |
| *** basha has joined #openstack-infra | 14:33 | |
| *** lttrl has joined #openstack-infra | 14:34 | |
| *** CaptTofu_ has joined #openstack-infra | 14:34 | |
| *** dansmith is now known as superdan | 14:34 | |
| *** atiwari has joined #openstack-infra | 14:36 | |
| *** zns has quit IRC | 14:37 | |
| *** andreykurilin_ has quit IRC | 14:38 | |
| *** andreykurilin_ has joined #openstack-infra | 14:39 | |
| *** zns has joined #openstack-infra | 14:40 | |
| *** cp16net_ has quit IRC | 14:41 | |
| *** xyang1 has joined #openstack-infra | 14:44 | |
| *** zz_gondoi is now known as gondoi | 14:45 | |
| *** zhiyan_ is now known as zhiyan | 14:46 | |
| *** james_li has joined #openstack-infra | 14:46 | |
| *** basha has quit IRC | 14:48 | |
| *** blamar has quit IRC | 14:48 | |
| anteaya | I'm answering a question on the ml about gerrit ports for firewalls, ensuring I have the right info here: https://gerrit-review.googlesource.com/Documentation/config-gerrit.html#sshd | 14:50 |
| anteaya | we use the defaults, 29418, 22 and 8080 for the web browser | 14:51 |
| *** _nadya_ has joined #openstack-infra | 14:51 | |
| anteaya | is that accurate? | 14:51 |
| *** om has joined #openstack-infra | 14:52 | |
| *** ildikov has quit IRC | 14:52 | |
| *** lttrl has quit IRC | 14:54 | |
| xyang1 | anteaya: question for you. When I apply for a service account, can I use a group email address? | 14:55 |
| openstackgerrit | Derek Higgins proposed a change to openstack-infra/config: Index the os-collect-config logs in tripleo jobs https://review.openstack.org/99950 | 14:55 |
| *** Longgeek_ has quit IRC | 14:57 | |
| anteaya | xyang1: use any email address where if an email comes in there will be a quick response | 14:58 |
| anteaya | xyang1: however you want to organize it from your end so that there is a quick response is entirely up to you | 14:58 |
| anteaya | so yes, if you want to use a group email, by all means do so | 14:58 |
| xyang1 | anteaya: thanks. a group email address will allow us to respond quicker | 14:59 |
| anteaya | fantastic | 14:59 |
| *** matjazp has joined #openstack-infra | 14:59 | |
| anteaya | that is the primary concern, quick dissemination of information, quick response | 14:59 |
| *** otherwiseguy has joined #openstack-infra | 15:00 | |
| xyang1 | anteaya: sure | 15:00 |
| *** lcostantino has quit IRC | 15:00 | |
| *** mrodden has quit IRC | 15:02 | |
| *** sarob_ has joined #openstack-infra | 15:03 | |
| anteaya | xyang1: thanks, which system is yours? | 15:04 |
| *** gondoi is now known as zz_gondoi | 15:04 | |
| *** changbl has quit IRC | 15:06 | |
| *** jaypipes is now known as leakypipes | 15:06 | |
| *** lcostantino has joined #openstack-infra | 15:07 | |
| *** sarob_ has quit IRC | 15:07 | |
| *** zhiyan is now known as zhiyan_ | 15:09 | |
| *** blamar has joined #openstack-infra | 15:10 | |
| jogo | sdague: just chimed into your stop hacking please email | 15:11 |
| *** blamar has quit IRC | 15:11 | |
| *** zul has quit IRC | 15:12 | |
| *** blamar has joined #openstack-infra | 15:12 | |
| *** zul has joined #openstack-infra | 15:12 | |
| *** zul has quit IRC | 15:14 | |
| *** zul has joined #openstack-infra | 15:15 | |
| *** mrodden has joined #openstack-infra | 15:16 | |
| *** changbl has joined #openstack-infra | 15:19 | |
| *** yfried_ has quit IRC | 15:22 | |
| *** andreykurilin_ has quit IRC | 15:23 | |
| *** sweston has quit IRC | 15:23 | |
| *** matjazp has quit IRC | 15:23 | |
| *** andreykurilin_ has joined #openstack-infra | 15:24 | |
| *** zul has quit IRC | 15:25 | |
| *** jlibosva has quit IRC | 15:25 | |
| *** timrc is now known as timrc-afk | 15:26 | |
| *** jistr has quit IRC | 15:27 | |
| *** jistr has joined #openstack-infra | 15:27 | |
| *** xchu has quit IRC | 15:29 | |
| *** annegentle has quit IRC | 15:29 | |
| arunkant | Can someone please approve it so that this change can merge. This has been already reviewed by infra-core and keystone-core and tried recheck..it did not re-trigger the build | 15:29 |
| arunkant | https://review.openstack.org/#/c/95842/ | 15:29 |
| anteaya | well you have a failing test | 15:30 |
| anteaya | so you need to look at the logs and see why the test failed first | 15:30 |
| anteaya | then when you have fixed the failing test with a new patch, or if you have found the bug number because if failed on a bug | 15:31 |
| *** e0ne has quit IRC | 15:31 | |
| mriedem | jogo: fungi: we should promote this https://review.openstack.org/#/c/99144/ since it's masking other timeout failures | 15:31 |
| anteaya | you can use `recheck bug <bug number>` | 15:31 |
| *** e0ne has joined #openstack-infra | 15:31 | |
| jogo | mriedem: works for me | 15:31 |
| mriedem | jogo: and how does one find out the pass rate of a given job? because i'd bet that (check|gate)-grenade-dsvm sucks right now | 15:32 |
| *** om has quit IRC | 15:32 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/config: Don't run large-ops test on stable/havana branches https://review.openstack.org/99750 | 15:33 |
| jogo | jogo.github.io/gate | 15:33 |
| jogo | which uses graphite | 15:33 |
| jogo | graphite.openstack.org | 15:33 |
| arunkant | anteaya: All of failed issues were related to connection . | 15:33 |
| jogo | and yes looks like your right the page I have uses moving average so spikes take a bit of time | 15:33 |
| anteaya | arunkant: great, can you find a bug for that? | 15:34 |
| mriedem | jogo: so looking at jogo.github.io/gate it looks like the two biggest gate failures right now are the grenade jobs right? | 15:35 |
| anteaya | arunkant: this might be a good place to start: https://bugs.launchpad.net/openstack/ | 15:35 |
| mriedem | failing around 25% of the time on average | 15:35 |
| *** e0ne has quit IRC | 15:36 | |
| jogo | mriedem: yeah, as I said it could be much worse since its a moving average | 15:36 |
| mriedem | this is the grenade bug i keep rechecking on https://bugs.launchpad.net/grenade/+bug/1315095 | 15:36 |
| xyang1 | anteaya: EMC's Cinder drivers | 15:36 |
| uvirtbot | Launchpad bug 1315095 in grenade "grenade nova network (n-net) fails to start" [Undecided,New] | 15:36 |
| mriedem | jogo: for whatever reason we don't have the *.failure logs from the grenade jobs getting archived | 15:36 |
| mriedem | according to the logs on the failure, they don't exist | 15:36 |
| *** wenlock_ has joined #openstack-infra | 15:37 | |
| mriedem | jogo: i.e. http://logs.openstack.org/44/99144/1/gate/gate-grenade-dsvm/3be9510/logs/devstack-gate-cleanup-host.txt | 15:37 |
| *** cp16net_ has joined #openstack-infra | 15:37 | |
| mriedem | 2014-06-13 14:31:00.112 | ls: cannot access /opt/stack/status/stack/*.failure: No such file or directory | 15:37 |
| *** CaptTofu_ has quit IRC | 15:37 | |
| mriedem | i'll update https://etherpad.openstack.org/p/gatetriage-june2014 | 15:38 |
| sdague | jogo: cool | 15:38 |
| sdague | jogo: so I think in future hacking major releases *have* to happen the week after the openstack release, or they hold to the next window | 15:38 |
| mtreinish | mriedem: I thought you pushed a fix for the *failure files grab | 15:39 |
| mriedem | mtreinish: i did | 15:39 |
| anteaya | xyang1: ah you don't have a system yet then? | 15:39 |
| mriedem | mtreinish: but looks like the files still don't exist for some reason | 15:40 |
| anteaya | xyang1: can we chat about the name before you submit your request? | 15:40 |
| anteaya | xyang1: for instance how large is EMC? do you think you will ever need to have more than one gerrit account? | 15:40 |
| anteaya | xyang1: and what is the name of the EMC component you are testing? | 15:41 |
| mtreinish | mriedem: hmm... | 15:41 |
| sdague | mtreinish: I'm not convinced the failure is always happening in a way that's recording | 15:42 |
| xyang1 | anteaya: sure. We are still setting them up. | 15:42 |
| xyang1 | anteaya: We'll need 4 accounts, one for each product | 15:43 |
| *** rwsu has joined #openstack-infra | 15:43 | |
| xyang1 | anteaya: VMAX cinder driver, VNX cinder driver, XIO cinder driver, and ViPR cinder driver | 15:43 |
| sdague | mriedem: man, this might be the elusive screen dropping stuff commands issue that we've seen before | 15:44 |
| sdague | which would suck | 15:44 |
| *** CaptTofu_ has joined #openstack-infra | 15:46 | |
| *** matjazp has joined #openstack-infra | 15:47 | |
| *** bknudson has joined #openstack-infra | 15:48 | |
| *** bknudson has quit IRC | 15:48 | |
| mriedem | sdague: yeah maybe | 15:49 |
| mriedem | https://bugs.launchpad.net/grenade/+bug/1315095/comments/2 | 15:49 |
| uvirtbot | Launchpad bug 1315095 in grenade "grenade nova network (n-net) fails to start" [Undecided,Confirmed] | 15:49 |
| mriedem | some notes there | 15:49 |
| *** bknudson has joined #openstack-infra | 15:49 | |
| sdague | there are other services failling to start. I actually have an earlier sanity check patch for this in my queue. I'll focus on fixing that today. | 15:49 |
| *** rlandy has quit IRC | 15:49 | |
| mriedem | yeah there are like 32 hits of 'fails to start' | 15:50 |
| *** matjazp has quit IRC | 15:50 | |
| anteaya | xyang1: okay so I will suggest 4 names `EMC VMAX CI`, `EMC VNX CI`, `EMC XIO CI`, and `EMC ViPR CI` | 15:50 |
| *** mbacchi has quit IRC | 15:50 | |
| anteaya | xyang1: let's leave cinder out of the names | 15:50 |
| anteaya | it is a shame they are all acroymns the all caps gets irritating | 15:51 |
| mriedem | WHY I DO NOT SEE THE ISSUE | 15:51 |
| anteaya | right like that | 15:51 |
| *** UtahDave has joined #openstack-infra | 15:51 | |
| anteaya | but acroymns are shorter | 15:51 |
| mriedem | i had a girlfriend in college that wrote her emails in all caps | 15:51 |
| mriedem | we didn't date for long... | 15:51 |
| anteaya | ha ha ha | 15:51 |
| mriedem | felt like i was being yelled at constantly | 15:51 |
| anteaya | yup | 15:51 |
| *** krtaylor has quit IRC | 15:52 | |
| anteaya | having mulitple ci systems yelling at you does'nt go over well | 15:52 |
| *** amcrn has quit IRC | 15:52 | |
| anteaya | but I can't stop companies from using acroymns | 15:52 |
| anteaya | xyang1: so that is what I suggest | 15:52 |
| *** pblaho has quit IRC | 15:52 | |
| xyang1 | anteaya: sure. thanks for the suggestion | 15:52 |
| anteaya | xyang1: and we will create usernames in lowercase with hyphens to match | 15:52 |
| anteaya | xyang1: it will help us process your request faster | 15:53 |
| *** CaptTofu_ has quit IRC | 15:53 | |
| *** zul has joined #openstack-infra | 15:53 | |
| xyang1 | anteaya: ok | 15:53 |
| anteaya | xyang1: so the usernames will be `emc-vmax-ci`, `emc-vnx-ci`, `emc-xio-ci`, and `emc-vipr-ci` | 15:53 |
| anteaya | and thanks | 15:53 |
| xyang1 | anteaya: we'll need to give you email address and ssh key before those names are accepted? | 15:54 |
| anteaya | we need the ssh key to create the accounts, so yes | 15:54 |
| anteaya | and put the email address in as well since that is the current process but that will soon change | 15:55 |
| xyang1 | anteaya: can we change the email address later after the account is created? | 15:55 |
| anteaya | I'm working on wikipage templates so you can list the emails yourself and modify them as you see fit | 15:55 |
| xyang1 | anteaya: ok, that's good | 15:55 |
| anteaya | xyang1: yes, the wikipages should be available soon | 15:56 |
| xyang1 | anteaya: thanks! | 15:56 |
| *** alkari has joined #openstack-infra | 15:56 | |
| anteaya | I'm working on them, I don't want to release them early since that would mean I would have to change 60 wikipages by hand after the fact | 15:56 |
| anteaya | xyang1: thanks for being available | 15:56 |
| xyang1 | anteaya: you are welcome. | 15:56 |
| anteaya | xyang1: irc is the easiest way to find me and ask me any questions since it saves me a ton of work after the fact | 15:57 |
| anteaya | :D | 15:57 |
| xyang1 | anteaya: you can find me on cinder irc too | 15:57 |
| anteaya | great | 15:57 |
| anteaya | thank you | 15:57 |
| xyang1 | anteaya: thanks | 15:58 |
| *** gyee has joined #openstack-infra | 15:58 | |
| *** basha has joined #openstack-infra | 15:58 | |
| *** basha has quit IRC | 15:58 | |
| *** basha has joined #openstack-infra | 15:59 | |
| *** zns has quit IRC | 16:00 | |
| *** freyes has quit IRC | 16:00 | |
| *** arnaud has joined #openstack-infra | 16:00 | |
| *** arnaud has quit IRC | 16:01 | |
| *** arnaud has joined #openstack-infra | 16:01 | |
| *** todd_dsm has joined #openstack-infra | 16:01 | |
| *** zns has joined #openstack-infra | 16:01 | |
| *** wenlock_ has quit IRC | 16:03 | |
| *** markwash has joined #openstack-infra | 16:04 | |
| *** dhellman_ has joined #openstack-infra | 16:05 | |
| *** todd_dsm has quit IRC | 16:05 | |
| *** dims_ has quit IRC | 16:07 | |
| *** krtaylor has joined #openstack-infra | 16:07 | |
| *** wenlock_ has joined #openstack-infra | 16:08 | |
| *** dims_ has joined #openstack-infra | 16:08 | |
| *** reed has joined #openstack-infra | 16:11 | |
| *** tcammann has quit IRC | 16:11 | |
| mordred | anteaya: we don't use 22 or 8080 for gerrit for people - only 29418 and 443 | 16:12 |
| *** marcoemorais has joined #openstack-infra | 16:12 | |
| *** zns has quit IRC | 16:12 | |
| *** tcammann has joined #openstack-infra | 16:12 | |
| *** derekh_ has quit IRC | 16:13 | |
| *** zns has joined #openstack-infra | 16:13 | |
| *** zul has quit IRC | 16:14 | |
| *** habib has quit IRC | 16:15 | |
| anteaya | mordred: thank you I will respond to the email | 16:15 |
| *** CaptTofu_ has joined #openstack-infra | 16:16 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: have realtime engine only search recent indexes https://review.openstack.org/99776 | 16:18 |
| *** lcostantino has quit IRC | 16:20 | |
| *** lcostantino has joined #openstack-infra | 16:20 | |
| *** zns has quit IRC | 16:22 | |
| *** zns has joined #openstack-infra | 16:24 | |
| reed | ttx, where is the classification you mention in programs.yaml? | 16:24 |
| reed | ttx: quoting your message to the mlist "I think they can go to your "other" category. That's how they are classified in programs.yaml." but I don't see such classification there | 16:25 |
| *** comstud is now known as bearhands | 16:25 | |
| *** leakypipes has quit IRC | 16:26 | |
| *** afazekas_ has quit IRC | 16:30 | |
| *** salv-orlando has joined #openstack-infra | 16:32 | |
| *** mbacchi has joined #openstack-infra | 16:34 | |
| *** ominakov has quit IRC | 16:35 | |
| *** viktors has quit IRC | 16:35 | |
| anteaya | paste.o.o 500'd on my again | 16:38 |
| trinaths | anteaya: hi | 16:39 |
| *** annegentlereally has quit IRC | 16:39 | |
| anteaya | hello trinaths | 16:39 |
| anteaya | trinaths: was there something on your mind? | 16:40 |
| trinaths | anteaya: yes. got a quick question. | 16:40 |
| anteaya | do share | 16:40 |
| trinaths | anteaya: regarding CI log repository | 16:40 |
| *** todd_dsm has joined #openstack-infra | 16:41 | |
| anteaya | well the logs aren't a repository | 16:41 |
| anteaya | a repository is something that is under version control | 16:41 |
| *** vponomaryov has left #openstack-infra | 16:41 | |
| trinaths | anteaya: what is the period do we need to store the logs.. (okay..) | 16:41 |
| anteaya | all of our projects are repositories since they are under git version control | 16:41 |
| trinaths | anteaya: mistaken.. corrected | 16:41 |
| *** Ryan_Lane has joined #openstack-infra | 16:42 | |
| *** Ryan_Lane has joined #openstack-infra | 16:42 | |
| anteaya | it was decided that 3rd party ci systems store their logs for one month | 16:42 |
| *** amcrn has joined #openstack-infra | 16:42 | |
| *** e0ne has joined #openstack-infra | 16:42 | |
| *** ramashri has joined #openstack-infra | 16:42 | |
| trinaths | anteaya: but, we have a change which spanned for 2 months and did not get merged. | 16:42 |
| trinaths | anteaya: (just for scenario) | 16:42 |
| anteaya | I'm all for storing the logs for longer | 16:43 |
| anteaya | but I got push back from other ci folks | 16:43 |
| trinaths | anteaya: if we delete the old logs, how can the owner check the old logs. | 16:43 |
| anteaya | I can't remember what room I was in at the summit but I do remember agreement on one month | 16:43 |
| anteaya | trinaths: very good point | 16:43 |
| trinaths | anteaya: but what will be the use, if we store logs for 'merged' changes?? | 16:43 |
| anteaya | if I had my way we would store all the logs forever | 16:43 |
| anteaya | trinaths: I don't understand your last question | 16:44 |
| trinaths | anteaya: let's have, a change is merged to master branch. now what will the use of the logs with CI | 16:45 |
| anteaya | that change got merged | 16:45 |
| trinaths | anteaya: yes | 16:45 |
| anteaya | the ci logs are the only logs that said what effect that change had on your driver or plugin | 16:45 |
| trinaths | anteaya: true | 16:45 |
| trinaths | anteaya: agree. but 3rd party CI tested it and posted success. | 16:46 |
| anteaya | so if there is a bug someone spots, they need to see the third party logs to see if it introduced the bug, or if it had an effect | 16:46 |
| anteaya | we revert code all the time | 16:46 |
| anteaya | since it can pass the tests and still introduce a bug | 16:46 |
| anteaya | if we have a race condidtion or something we don't test for | 16:47 |
| *** zul has joined #openstack-infra | 16:47 | |
| *** e0ne has quit IRC | 16:47 | |
| trinaths | anteaya: okay, good. but then our one month linmit to logs will not work in this scenario. | 16:47 |
| anteaya | and the way to find it is to work backwards | 16:47 |
| anteaya | I'm not arguing with your logic | 16:47 |
| trinaths | s/linmit/limit | 16:47 |
| anteaya | but I am saying that a limit was agreed to | 16:47 |
| anteaya | and the limit was one month | 16:47 |
| trinaths | anteaya: yes. but just want to make my thing clear | 16:47 |
| anteaya | save them all if you can, I would really apprecaite that | 16:47 |
| anteaya | yes, you clearly have identified a failure in the logic of the limit | 16:48 |
| anteaya | I see that | 16:48 |
| anteaya | and I am not disputing that | 16:48 |
| trinaths | anteaya: okay. | 16:48 |
| devananda | sdague: did your change to make gate-tempest-dsvm-virtual-ironic non-voting put ironic and diskimage-builder in a separate merge queue, or did something else do that? or am I smoking crack? | 16:48 |
| anteaya | but in my role of ensuring compliance for over 50 systems, a limit was agreed to at the last summit - one month | 16:48 |
| *** dangers_away is now known as dangers | 16:48 | |
| anteaya | trinaths: if you wish the limit to be changed, do start a thread on the ml stating your position | 16:49 |
| *** thedodd has quit IRC | 16:49 | |
| anteaya | if the community agrees that third party ci logs should be stored for a duration different than one month then that is what I will communicate and ensure | 16:49 |
| *** saper has joined #openstack-infra | 16:49 | |
| trinaths | anteaya: okay. one month storage of logs is good. but when the change is in review for 2 months, then, owner of that change, may not see the old logs with 3rd party CI | 16:50 |
| anteaya | yes, that is the current situation, you are correct | 16:50 |
| *** dizquierdo has quit IRC | 16:51 | |
| trinaths | anteaya: also, in the scenario you said, reagarding the bugs and backward checks, too when we loose the logs after 1 month, it will not help the backward check. | 16:53 |
| *** hashar has quit IRC | 16:53 | |
| trinaths | anteaya: I felt storage as a issue here, hence I shared with you. | 16:54 |
| anteaya | trinaths: I'm grateful we are discussing it | 16:54 |
| anteaya | yes, storage is the limiting factor in log retention | 16:55 |
| anteaya | for us as well | 16:55 |
| anteaya | we keep 6 months of logs, we would like to keep more, but we have too many logs | 16:55 |
| trinaths | anteaya: true. agree | 16:56 |
| reed | trinaths, the issue you raise is important, probably a thread on the mlist is good | 16:56 |
| *** wenlock_ has quit IRC | 16:56 | |
| *** dhellman_ has quit IRC | 16:56 | |
| reed | it's a topic related somewhat to the ongoing discussion about compute quota | 16:56 |
| trinaths | reed: may I mail this in infra list | 16:57 |
| reed | trinaths, openstack-dev seems more appropriate | 16:57 |
| *** bknudson has left #openstack-infra | 16:57 | |
| trinaths | reed: okay | 16:57 |
| anteaya | trinaths: yes use the tag [3rd party] in the email subject line and email to -dev | 16:57 |
| anteaya | it is okay to cc the infra list too if you want | 16:58 |
| *** hemna_ is now known as hemna | 16:58 | |
| anteaya | but the conversation will take place on -dev | 16:58 |
| *** sweston has joined #openstack-infra | 16:58 | |
| *** james_li has quit IRC | 16:59 | |
| trinaths | anteaya: okay. preparing the mail. | 16:59 |
| anteaya | kk | 17:00 |
| *** fbo is now known as fbo_away | 17:01 | |
| fungi | mriedem: jogo: does 99144 still merit promoting>? | 17:02 |
| clarkb | o/ | 17:03 |
| clarkb | fungi: sdague: after talking to dstufft last night I am much more on board with spinning up an ES 07 and rotating 01 out | 17:04 |
| *** dims_ has quit IRC | 17:04 | |
| clarkb | fungi: what do you think? | 17:04 |
| vishy | so I’m having trouble seeing what went wrong here: http://logs.openstack.org/54/93754/6/check/check-grenade-dsvm/7f76538/console.html | 17:04 |
| *** sarob_ has joined #openstack-infra | 17:04 | |
| vishy | it looks like ceilometer-dbsync randomly died in the middle | 17:04 |
| fungi | clarkb: oh, excellent. i missed a lot of scrollback (didn't actually sleep at all last night, stayed up until dawn migrating the last of my servers into rackspace)... what's the tl;dr? | 17:04 |
| *** todd_dsm has quit IRC | 17:05 | |
| *** harlowja_away is now known as harlowja | 17:06 | |
| clarkb | fungi: wow thats dedication. tl;dr is pypi gluster servers ran into similar issues and they worked around it by spinning up new nodes | 17:06 |
| clarkb | fungi: so 1 it makes me feel less crazy about blaming the cloud and 2 that work around is known to work with a small sample size | 17:06 |
| *** shayneburgess has joined #openstack-infra | 17:06 | |
| fungi | clarkb: not so much dedication as deadline. junk removal was scheduled to come today during lunchtime so i could get it all hauled off | 17:07 |
| clarkb | vishy: it says the main setup script failed so http://logs.openstack.org/54/93754/6/check/check-grenade-dsvm/7f76538/logs/old/devstacklog.txt.gz should be the place to look | 17:07 |
| clarkb | vishy: but I don't see anything there either | 17:07 |
| *** markwash has quit IRC | 17:08 | |
| fungi | clarkb: so the pypi crew saw i/o issues isolated to a single node in their cluster, and it went away when they replace the problem node? that definitely sounds like a misbehaving neighbor if so | 17:08 |
| shayneburgess | quick question, if someone can help. I am trying to submit my first patch for review and I get the following error from “git review”: fatal: ICLA contributor agreement requires current contact information. | 17:08 |
| shayneburgess | Please review your contact information: | 17:08 |
| shayneburgess | https://review.openstack.org/#/settings/contact | 17:08 |
| shayneburgess | fatal: Could not read from remote repository. | 17:08 |
| *** sarob_ has quit IRC | 17:08 | |
| clarkb | fungi: yes and rax doesn't guarantee IOPS unless you spend lots of money or some such | 17:08 |
| vishy | clarkb: I added it to here https://bugs.launchpad.net/ceilometer/+bug/1221580 since it looks similar | 17:08 |
| trinaths | anteaya: done. | 17:08 |
| uvirtbot | Launchpad bug 1221580 in ceilometer "devstack's call of ceilometer-dbsync gets stuck after migration 6 -> 7" [Medium,Triaged] | 17:08 |
| *** lcostantino has quit IRC | 17:08 | |
| clarkb | fungi: and we seem to feel the same problems | 17:08 |
| shayneburgess | When I go to that url and update my information it gives a server error | 17:09 |
| clarkb | shayneburgess: if you open that link and update your contact info does the error go away? | 17:09 |
| fungi | clarkb: makes total sense in that case, yes | 17:09 |
| shayneburgess | clarkb: It gives the following error: Server Error Cannot store contact information | 17:09 |
| clarkb | shayneburgess: ah ok. The thing to check in that case is that you are an openstack foundation member, and that your primary email for that account matches the primary email for your gerrit account | 17:10 |
| *** pelix has quit IRC | 17:10 | |
| mestery | anteaya: Sent the email to openstack-dev around 3rd party testing, sorry it took a few days longer than I originally thought. | 17:10 |
| shayneburgess | ok. i think that both of those are true. Verifying now | 17:10 |
| mestery | anteaya: I put deadlines in there too. | 17:10 |
| anteaya | mestery: awesome thank you | 17:10 |
| mestery | anteaya: Also highlighted the weekly third party meeting you host, hope to get more people to attend that and share experiences and askq uestions. | 17:11 |
| shayneburgess | clarkb: I have validated the same email address for both accounts: shayne.burgess@hp.com | 17:11 |
| clarkb | fungi: great, I will start spinning up a new es07 node shortly. did that puppet change merge to not puppet 07when it first comes up? | 17:11 |
| clarkb | fungi: then once everything is happy on 07 I will kill 01 but keep it around so that we can run bonnie++ and do other sanity checking | 17:12 |
| *** todd_dsm has joined #openstack-infra | 17:12 | |
| fungi | clarkb: i believe so... looking | 17:12 |
| *** mmaglana has joined #openstack-infra | 17:12 | |
| *** rcarrill` has joined #openstack-infra | 17:13 | |
| anteaya | mestery: great thank you | 17:13 |
| *** wenlock_ has joined #openstack-infra | 17:13 | |
| clarkb | shayneburgess: I just updated my contact info to make sure it wasn't something like a dead server and it updated for me | 17:14 |
| *** Ryan_Lane1 has joined #openstack-infra | 17:14 | |
| clarkb | shayneburgess: are you sure the account at https://www.openstack.org/profile/ matches gerrit? | 17:14 |
| clarkb | shayneburgess: note it must be the primary email addresses | 17:14 |
| *** ramashri has quit IRC | 17:14 | |
| trinaths | anteaya: got my mail ? | 17:15 |
| anteaya | trinaths: I have many emails, and yes I read yours | 17:15 |
| *** rcarrillocruz has quit IRC | 17:16 | |
| anteaya | you outline your understanding but I am foggy on the actual question you are asking | 17:16 |
| shayneburgess | clarkb: checking | 17:16 |
| anteaya | let it steep for the weekend | 17:16 |
| shayneburgess | clarkb: that fixed it. I had the wrong email address on this form: https://www.openstack.org/profile/ | 17:17 |
| trinaths | anteaya: okay. is my mail not clear? | 17:17 |
| shayneburgess | thanks for your help! | 17:17 |
| *** rcarrill` has quit IRC | 17:18 | |
| anteaya | trinaths: i think it is more of a cultural thing | 17:19 |
| *** arnaud has quit IRC | 17:19 | |
| anteaya | trinaths: my sense is you come from a culture that uses the third person when discussing topics and I come from a culture where I use the first person | 17:19 |
| anteaya | trinaths: so it is more of a cultural difference than a lack of facts | 17:20 |
| anteaya | we are learning to communicate with each other, you and I | 17:20 |
| *** davidlenwell is now known as davidlenwell_ | 17:20 | |
| *** davidlenwell_ is now known as davidlenwell | 17:20 | |
| anteaya | and patience on both of our sides helps | 17:20 |
| anteaya | thank you for your patience with me | 17:20 |
| trinaths | anteaya: okay | 17:21 |
| *** rcarrillocruz has joined #openstack-infra | 17:21 | |
| *** ramashri has joined #openstack-infra | 17:23 | |
| *** markwash has joined #openstack-infra | 17:24 | |
| trinaths | happy weekend. bye all | 17:24 |
| *** trinaths has quit IRC | 17:24 | |
| mriedem | fungi: sdague: i'd say yeah https://review.openstack.org/#/c/99144/ merits promoting since it's masking other failures, but i'm not pushing it | 17:25 |
| *** _nadya_ has quit IRC | 17:26 | |
| *** _nadya_ has joined #openstack-infra | 17:26 | |
| clarkb | fungi: `cinder absolute-limits` and `cinder quota-show` are failing for me. How did you check that quota? | 17:26 |
| fungi | mriedem: too late. i already promoted it a little while ago and it's ~45 minutes from merging :/ | 17:27 |
| mriedem | fungi: works for me :) was at lunch | 17:27 |
| *** Guest47184 has quit IRC | 17:28 | |
| *** melwitt has joined #openstack-infra | 17:29 | |
| *** markmcclain has joined #openstack-infra | 17:30 | |
| *** marcoemorais has quit IRC | 17:30 | |
| *** marcoemorais has joined #openstack-infra | 17:30 | |
| *** praneshp has joined #openstack-infra | 17:31 | |
| * clarkb tries upgrading cinder client in a different venv | 17:31 | |
| *** amcrn_ has joined #openstack-infra | 17:34 | |
| clarkb | fungi: and you are comfortable adding another 1TB volume to a rapidly shrinking volume quota? | 17:34 |
| clarkb | fungi: looks like we have ~2TB free currently/ | 17:34 |
| clarkb | (new cinder fixed the errors for me) | 17:35 |
| *** amcrn has quit IRC | 17:36 | |
| mordred | clarkb: woot | 17:36 |
| clarkb | mordred: do you want a tl;dr on es situation? or have you been able to follow scrollback | 17:36 |
| mordred | clarkb: I think I've been able to follow the scrollback | 17:36 |
| clarkb | cool | 17:36 |
| sdague | devananda: I don't know about the non voting change doing that or not | 17:37 |
| anteaya | so what is our current direction for addressing the gate distress? | 17:37 |
| clarkb | I am ready to boot a node and create a volume if other folks are generally happy with that plan given the quota situation | 17:37 |
| mordred | ++ | 17:37 |
| anteaya | clarkb: go you | 17:38 |
| anteaya | clarkb: that is for the es situation and won't affect the gate situation, is that correct? | 17:39 |
| clarkb | correct | 17:39 |
| *** markmc has quit IRC | 17:39 | |
| anteaya | kk, happy es gives waiting devs something to do | 17:39 |
| clarkb | anteaya: I haevn't caught up on scrollback myself. Is gate situation the lack of nodes? | 17:40 |
| *** markwash has quit IRC | 17:40 | |
| mordred | clarkb: that situation has to do with problems deleting, right? | 17:41 |
| anteaya | yes | 17:41 |
| anteaya | more headed for delete state all the time | 17:41 |
| sdague | yeh, the deleting trend line means we'll be at 100% deleting by mid next week I think | 17:41 |
| anteaya | there hasnt' been much conversation around it today since you just showed up and fungi has been busy most of the morning | 17:41 |
| mordred | so - this is a problem with _both_ clouds? | 17:42 |
| anteaya | rax | 17:42 |
| sdague | mordred: I don't know | 17:42 |
| anteaya | this is a rax cloud issue | 17:42 |
| mordred | yah. so at some point I'd expect rax to be at 100% deleting and our only source of nodes to be hp, yeah/ | 17:42 |
| anteaya | gilliard: was in earlier asking if he could access nodepool info so he could assess things from his end | 17:42 |
| *** ildikov has joined #openstack-infra | 17:43 | |
| sdague | but I'd love clarkb to stay focussed on ES because we're pretty handicapped on addressing the right race bugs until that's working again | 17:43 |
| mordred | anteaya: ah - ok. yes- I can get him as much info as he wants | 17:43 |
| mordred | sdague: ++ | 17:43 |
| *** zul has quit IRC | 17:43 | |
| anteaya | mordred: he wants a wall with nodepool updates on it | 17:43 |
| *** zul has joined #openstack-infra | 17:43 | |
| sdague | and we're down 3 infra core today, as it's a holiday in russia, fungi is moving, and jeblair is still out | 17:43 |
| *** maxbit has quit IRC | 17:43 | |
| *** todd_dsm has quit IRC | 17:43 | |
| anteaya | at least that is what I got from what he posting in scrollback | 17:43 |
| sdague | so mordred, if you could look into the deleting issue, that would be huge | 17:43 |
| mordred | anteaya: can you expand on that? | 17:44 |
| mordred | anteaya: "wall with nodepool updates" ? | 17:44 |
| *** zzelle has joined #openstack-infra | 17:44 | |
| anteaya | mordred: I will find the statement in the logs | 17:44 |
| sdague | mordred: today nodepool status is something only infra core can get, correct/ | 17:44 |
| clarkb | mordred: I think tail -f of the logs | 17:44 |
| sdague | ? | 17:44 |
| clarkb | sdague: yes bceause of leakage of stuff | 17:44 |
| anteaya | 2014-06-13T09:50:36 <gilliard> Hi all - is there something wrong with nodepool? Looks like >50% instances are "deleting" | 17:44 |
| clarkb | there was that big discussion with morganfainberg about how we dump data into logs we shouldn't | 17:45 |
| sdague | clarkb: ok. | 17:45 |
| mordred | anteaya: gotcha | 17:45 |
| anteaya | 2014-06-13T10:06:09 <gilliard> I haven't any way to investigate nodepool itself but if anyone suspects problems with HP instances just give me a shout :) | 17:45 |
| clarkb | ok booting elasticsearch07 now | 17:45 |
| anteaya | 2014-06-13T10:29:58 <gilliard> sdague: OK. Do you have a graphite link that splits up the nodepool data by cloud? | 17:45 |
| mordred | anteaya: k. from the graphs I'm looking at, hp looks healthy | 17:45 |
| anteaya | I'll stop now, but that is the idea | 17:45 |
| mordred | yah | 17:45 |
| *** gyee has quit IRC | 17:45 | |
| *** todd_dsm has joined #openstack-infra | 17:46 | |
| anteaya | yeah, I hadn't heard anything from hp | 17:46 |
| anteaya | fortunately | 17:46 |
| *** markwash has joined #openstack-infra | 17:47 | |
| *** chuck__ has joined #openstack-infra | 17:47 | |
| *** markwash has quit IRC | 17:47 | |
| *** zul has quit IRC | 17:47 | |
| *** sweston has quit IRC | 17:47 | |
| fungi | clarkb: yeah, we have just under 2tb available, so if we need to temporarily dip a tb into that for a node swap over a weekend and into next week it's not a problem i think | 17:48 |
| fungi | sdague: yeah i hate that i'm not being more helpful. i'll be putting in some serious extra effort once i'm settled in at the far end of the move | 17:49 |
| clarkb | fungi: no problem. and thanks | 17:49 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: [WIP] Added search interface https://review.openstack.org/99975 | 17:50 |
| sdague | fungi: no problem, life is allowed to happen :) | 17:50 |
| sdague | I was mostly just prodding mordred to take ownership on the delete side, as we are definitely down a lot of capacity now | 17:51 |
| *** todd_dsm has quit IRC | 17:51 | |
| *** markwash has joined #openstack-infra | 17:51 | |
| anteaya | yes, there are times when we need mordred and now is one of them | 17:51 |
| fungi | is graphite struggling or is this just the crappy wireless access point i switched to in the past couple hours? | 17:53 |
| fungi | i guess it's just me. i can directly load stuff from graphite but the images embedded in the zuul status page are teh broken for me | 17:53 |
| fungi | and now they're suddenly better again. whatever | 17:54 |
| *** nati_ueno has joined #openstack-infra | 17:54 | |
| fungi | but yeah, the delete accumulation is getting pretty bad. i can try to burn down some of those but not sure if it will be more than a very temporary relief | 17:54 |
| mordred | sdague, fungi: I saw discussion with comstud in here yesterday on delete - is there an action item from our side we need to be doing? | 17:55 |
| *** praneshp has quit IRC | 17:55 | |
| *** sweston has joined #openstack-infra | 17:56 | |
| fungi | mordred: i think the main takeaway there is that there's no point in retrying a delete in nodepool unless the node is still showing as "active" in nova (as nova delete calls on an instance which is in deleting or error state are entirely ignored and therefore wasted effort) | 17:56 |
| sdague | someone needs to follow up with rax and actually get them to cleanup all the errored deleting nodes | 17:56 |
| sdague | because it can't be done by us | 17:57 |
| sdague | there is also possibly a nova fix for this | 17:57 |
| sdague | but that's long term | 17:57 |
| fungi | yeah, for the "nova can't delete nodes which went into error state after the delete call" bug | 17:57 |
| sdague | right | 17:57 |
| sdague | but that has to go through code review, land, and get into rax cloud. So that's not a near term fix | 17:58 |
| fungi | we currently have 353 nodes which nodepool believes it has been trying to delete for more than one hour. i'm going to set a parallel fire on these to try and clear away as many as i can | 17:58 |
| clarkb | ok volume created, node booted. time to do the cinder nova + lvm dance | 17:59 |
| clarkb | this should be fun | 17:59 |
| sdague | fungi: all in rax? | 17:59 |
| sdague | or are some in hp? | 17:59 |
| fungi | sdague: most of rax is in that state, but also some in hp | 18:00 |
| fungi | we already know that some percentage of nova delete calls simply fall on the floor but work on the second (third, fourth, fifth) retry | 18:00 |
| reed | it's Community Office Hour time :) if you want to talk, I'm here | 18:00 |
| *** _nadya_ has quit IRC | 18:00 | |
| clarkb | "attaching" | 18:00 |
| clarkb | woot in use | 18:00 |
| *** slagle has joined #openstack-infra | 18:01 | |
| *** melwitt has quit IRC | 18:01 | |
| *** melwitt has joined #openstack-infra | 18:01 | |
| *** todd_dsm has joined #openstack-infra | 18:02 | |
| mordred | clarkb: woot | 18:02 |
| *** praneshp has joined #openstack-infra | 18:03 | |
| clarkb | fungi: mordred: looks like I need to vgcreate main. Are we using any special options on that like zeroing? | 18:04 |
| clarkb | I don't think we need zeroing for elasticsearch (data is public anyways) | 18:04 |
| mordred | clarkb: I don't think so? | 18:05 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: [WIP] Added search interface https://review.openstack.org/99975 | 18:05 |
| fungi | clarkb: i haven't been, no | 18:05 |
| *** todd_dsm has quit IRC | 18:06 | |
| *** todd_dsm has joined #openstack-infra | 18:06 | |
| *** annegentlereally has joined #openstack-infra | 18:08 | |
| *** chuck__ is now known as zul | 18:09 | |
| *** zul has quit IRC | 18:09 | |
| *** zul has joined #openstack-infra | 18:09 | |
| clarkb | Insufficient free extents (262143) in volume group main: 262144 required <- I only need one more free extent :/ | 18:09 |
| clarkb | I guess I lvcreate -l | 18:10 |
| *** zehicle_at_dell has joined #openstack-infra | 18:10 | |
| *** CaptTofu_ has quit IRC | 18:12 | |
| *** gyee has joined #openstack-infra | 18:13 | |
| fungi | clarkb: yeah if i'm using the entire pv i just call out the extent count explicitly | 18:13 |
| *** annegentlereally has quit IRC | 18:14 | |
| fungi | clarkb: though you can also do 100%FREE notation for size i think | 18:14 |
| mordred | clarkb: can't you just get another extent from wal-mart? | 18:14 |
| clarkb | yeah I used -l100%FREE | 18:14 |
| *** arnaud has joined #openstack-infra | 18:14 | |
| fungi | mordred: they only sell extents in thousand-packs there | 18:14 |
| *** mmaglana has quit IRC | 18:15 | |
| clarkb | waiting for a filesystem to be created | 18:15 |
| *** maxbit has joined #openstack-infra | 18:15 | |
| sdague | clarkb: the optimized version of the bot is landed | 18:16 |
| clarkb | sdague: thanks | 18:16 |
| sdague | hopefully that helps a bit | 18:16 |
| *** CaptTofu_ has joined #openstack-infra | 18:17 | |
| clarkb | in theory ES can start rebalancing soonish. Just need filesystem. reboot. merge puppet change and puppet all the things | 18:17 |
| clarkb | I should write that puppet change | 18:17 |
| *** ihrachyshka has joined #openstack-infra | 18:17 | |
| *** annegentlereally has joined #openstack-infra | 18:17 | |
| *** james_li has joined #openstack-infra | 18:18 | |
| *** amcrn_ has quit IRC | 18:18 | |
| *** flaper87 is now known as flaper87|afk | 18:20 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Add elasticsearch07 https://review.openstack.org/99980 | 18:20 |
| clarkb | I need to create DNS records before that should merge | 18:20 |
| clarkb | oh hey the launch script does the /opt dance now | 18:22 |
| clarkb | there is fun stuff in fstab | 18:22 |
| clarkb | I am going to remove the swap entry in fstab though to be like the other nodes | 18:22 |
| sdague | clarkb: fyi, I'm going to pumkin soon. Need to drive to VT for family activities for the weekend. Any other things you will need from me before doing so? | 18:24 |
| jerryz | fungi: about log retention, could we keep logs of a failed job longer and successful shorter? just curious. | 18:26 |
| anteaya | jerryz: is this in response to the ml thread? | 18:27 |
| anteaya | there is a ml thread about log retention for third party ci systems | 18:28 |
| anteaya | is your question motivated by that thread? | 18:28 |
| jerryz | anteaya: just provoked by it | 18:28 |
| anteaya | great | 18:28 |
| clarkb | sdague: I don't think so, have fun | 18:28 |
| anteaya | well to keep the conversation all in one place, it would be best if you could reply to the thread | 18:28 |
| clarkb | sdague: at this point its massaging es onto a new host | 18:28 |
| anteaya | right now it is log retention for one month for third party ci systems regardless of log build outcome | 18:29 |
| sdague | clarkb: cool | 18:29 |
| anteaya | if you want to discuss that, it is best if you reply to the email so that trianth gets a feeling for how other people feel | 18:29 |
| sdague | did you do an ssd volume? | 18:29 |
| jerryz | anteaya: not necessarily for 3rd party CI, but all the logs. | 18:29 |
| sdague | or is that a later attempt of black magic | 18:29 |
| clarkb | sdague: later attempt | 18:30 |
| jerryz | anteaya: because not a review, but also a bug may stay unattended or under discussion for a long time | 18:30 |
| clarkb | sdague: I want to make a like for like copy to reduce variables | 18:30 |
| mordred | clarkb: at some point, not right now, I'd love to capture "massaging ES onto a new host" | 18:30 |
| anteaya | jerryz: well the thread is addressing third party ci systems log retention | 18:31 |
| anteaya | openstack log retention is 6 months | 18:31 |
| clarkb | mordred: its pretty straight forward, boot node, edit DNS records, attach volume and do all of the necessary (mkfs, fstab, create mount point and user:group for that location), update firewalls, run puppet | 18:31 |
| clarkb | mordred: iptables and volume stuff being the clunkiest | 18:31 |
| mordred | nod | 18:31 |
| clarkb | ok rebooting es07 now to make sure everything comes up | 18:32 |
| jerryz | anteaya: so now third party CI can archive logs to static.o.o ? i still remembered they archived in their own locations | 18:32 |
| anteaya | jerryz: no | 18:32 |
| anteaya | jerryz: third party ci systems archive to the url they provide in their gerrit comments | 18:32 |
| anteaya | where did you get they archive to static.o.o based on what I just said? | 18:33 |
| anteaya | I'm confused | 18:33 |
| *** _nadya_ has joined #openstack-infra | 18:33 | |
| jerryz | anteaya: i thought 3rd party ci maintain their own log server | 18:33 |
| anteaya | yes they do | 18:34 |
| anteaya | and the url for their log server is provided in every gerrit comment | 18:34 |
| *** ociuhandu has quit IRC | 18:34 | |
| jerryz | anteaya: where does the one month period apply, then? | 18:35 |
| *** todd_dsm has quit IRC | 18:35 | |
| anteaya | for the log retention of the third party ci systems | 18:35 |
| *** talluri has joined #openstack-infra | 18:35 | |
| clarkb | fungi: mordred ok es07 is rebooted, DNS records created. anyone else want to login to that machine and check things before I self approve https://review.openstack.org/#/c/99980/ | 18:36 |
| clarkb | also only 20 changes from 100k | 18:36 |
| anteaya | so if I click the link of your gerrit comments I need to be able to access those logs on your server for 30 days | 18:36 |
| jerryz | anteaya: so 3rd party CI need to archive a minimum of one month of logs | 18:36 |
| anteaya | yes | 18:36 |
| anteaya | exactly right | 18:36 |
| anteaya | if you want to archive more, that is great | 18:37 |
| anteaya | but right now we only require one month | 18:37 |
| anteaya | and for me one month == 30 days | 18:37 |
| jerryz | anteaya: and trinath wants it to be longer | 18:38 |
| anteaya | and trianth has a ml post up to discuss having a longer limit | 18:38 |
| anteaya | yes | 18:38 |
| anteaya | that is my understanding of his position | 18:38 |
| clarkb | mordred: fungi: I am going to find breakfast/lunch and give you all a few minutes to look at things if yo uare able | 18:38 |
| anteaya | and for the limit to change requires input from the community | 18:38 |
| fungi | clarkb: i'll poke around | 18:38 |
| clarkb | thanks | 18:38 |
| openstackgerrit | A change was merged to openstack-dev/hacking: Fix a typo in HACKING.rst https://review.openstack.org/98435 | 18:40 |
| jerryz | anteaya: sure. my advise still kinda solve the conflict between more data and less storage. unless there are still some values in successful logs i can't think of. | 18:40 |
| sdague | clarkb: yeh, I'm surprised someone hasn't botted to snag it :) | 18:40 |
| clarkb | sdague: I will shame anyone if they bot it | 18:40 |
| clarkb | er shame that someone | 18:40 |
| anteaya | jerryz: advise on the mailing list | 18:41 |
| sdague | I think someone did for the last counter roll over | 18:41 |
| fungi | sdague: i'm more interested in when we hit the one millionth review | 18:41 |
| anteaya | jerryz: but keep in mind the limit will be teh same regardless of log build outcome | 18:41 |
| fungi | we're just about 10% of the way there! | 18:41 |
| anteaya | we aren't having one limit for success and a different limit for failure | 18:41 |
| *** dims_ has joined #openstack-infra | 18:41 | |
| reed | gentle ping for this review https://review.openstack.org/99481 | 18:42 |
| *** _nadya_ has quit IRC | 18:45 | |
| *** tkelsey has quit IRC | 18:45 | |
| *** shayneburgess has quit IRC | 18:46 | |
| *** maxbit has quit IRC | 18:47 | |
| *** otherwiseguy has quit IRC | 18:49 | |
| *** chuck__ has joined #openstack-infra | 18:49 | |
| *** reaper has quit IRC | 18:49 | |
| openstackgerrit | K Jonathan Harker proposed a change to openstack-infra/infra-specs: Put the puppet modules in their own projects/repos https://review.openstack.org/99990 | 18:49 |
| fungi | clarkb: the machine and the change to bring it into service look like they're in good shape. here's hoping this gets elasticsearch caught back up | 18:51 |
| *** zul has quit IRC | 18:51 | |
| * sdague would do a happy dance if that happens | 18:52 | |
| anteaya | oh my look at all that blue in the node graph | 18:52 |
| sdague | anteaya: yeh, I was about to say | 18:52 |
| anteaya | all those 'in use' nodes | 18:53 |
| sdague | yay capacity | 18:53 |
| * anteaya does a happy dance | 18:53 | |
| fungi | that's my mass delete bearing fruit, apparently | 18:53 |
| *** _nadya_ has joined #openstack-infra | 18:54 | |
| anteaya | woooooo | 18:54 |
| *** ihrachyshka has quit IRC | 18:54 | |
| anteaya | fungi: thanks for your magic scripts | 18:54 |
| *** CaptTofu_ has quit IRC | 18:55 | |
| *** CaptTofu_ has joined #openstack-infra | 18:55 | |
| clarkb | fungi: thanks I am going to go ahead and start applying things then | 18:57 |
| *** todd_dsm has joined #openstack-infra | 18:58 | |
| *** ihrachyshka has joined #openstack-infra | 18:58 | |
| clarkb | ok pupept change is approved | 18:58 |
| lifeless | clarkb: you aware of the tripleoci downness? | 19:00 |
| lifeless | \\ is it still ongoing? | 19:00 |
| *** dims has joined #openstack-infra | 19:00 | |
| clarkb | I am not aware. sorry | 19:00 |
| clarkb | I was asked earlier to focus on elasticsearch | 19:00 |
| *** shayneburgess has joined #openstack-infra | 19:01 | |
| *** dprince has quit IRC | 19:01 | |
| fungi | lifeless: when derekh asked about it earlier, it appeared that all nodes in the red hat region were in a delete state in nodepool, and it had been hours since nodepool had retried tried to delete them (saw some tracebacks returned from nova in the log which i summarized in channel) | 19:01 |
| *** cody-somerville has quit IRC | 19:01 | |
| bnemec | Hey infra folks, the dib-utils project has been created and according to the instructions I'm following I need to ask someone here to add me to dib-utils-core so I can get the review groups set up correctly. | 19:01 |
| fungi | bnemec: done | 19:02 |
| *** dims_ has quit IRC | 19:03 | |
| bnemec | fungi: Awesome, thanks! | 19:03 |
| *** mbacchi has quit IRC | 19:06 | |
| openstackgerrit | A change was merged to openstack-dev/hacking: Mark hacking as being a universal wheel https://review.openstack.org/99528 | 19:07 |
| *** alexpilotti_ has joined #openstack-infra | 19:08 | |
| *** e0ne has joined #openstack-infra | 19:08 | |
| *** alexpilotti has quit IRC | 19:11 | |
| *** alexpilotti_ is now known as alexpilotti | 19:11 | |
| openstackgerrit | A change was merged to openstack-infra/reviewstats: Oslo project updates https://review.openstack.org/99184 | 19:12 |
| *** mmaglana has joined #openstack-infra | 19:13 | |
| *** _nadya_ has quit IRC | 19:14 | |
| openstackgerrit | Jaroslav Henner proposed a change to openstack-infra/jenkins-job-builder: add presend-script to email-ext https://review.openstack.org/99994 | 19:14 |
| *** otherwiseguy has joined #openstack-infra | 19:20 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add elasticsearch07 https://review.openstack.org/99980 | 19:22 |
| *** matjazp has joined #openstack-infra | 19:23 | |
| clarkb | woot that will be applied to the various nodes over the next few minutes, I will run puppet by hand on 07 once firewalls are updated | 19:24 |
| openstackgerrit | mark mcclain proposed a change to openstack-infra/reviewstats: add neutron-specs https://review.openstack.org/100001 | 19:25 |
| *** chuck__ has quit IRC | 19:26 | |
| *** james_li has quit IRC | 19:26 | |
| clarkb | we have hit 100k | 19:28 |
| clarkb | greghaynes is to blame | 19:28 |
| jesusaurus | greghaynes: you were sitting on your change waiting for that, weren't you? | 19:30 |
| *** james_li has joined #openstack-infra | 19:33 | |
| greghaynes | :) | 19:34 |
| greghaynes | mebbe | 19:34 |
| *** talluri has quit IRC | 19:35 | |
| *** talluri has joined #openstack-infra | 19:35 | |
| *** talluri has quit IRC | 19:40 | |
| *** Ryan_Lane1 has quit IRC | 19:40 | |
| *** todd_dsm has quit IRC | 19:43 | |
| *** nati_ueno has quit IRC | 19:43 | |
| *** zns has quit IRC | 19:44 | |
| mordred | goooooooooooooooool | 19:44 |
| lifeless | bnemec: fungi actually dib-utils should not have its own review group, it should use tripleo-core and tripleo-ptl | 19:45 |
| *** markwash has quit IRC | 19:45 | |
| *** annegentlereally has quit IRC | 19:45 | |
| bnemec | lifeless: It does, but it's configured like diskimage-builder where diskimage-builder-core includes tripleo-core. | 19:46 |
| clarkb | I am runnin gpuppet on 07 now | 19:46 |
| *** todd_dsm has joined #openstack-infra | 19:47 | |
| *** bookwar has quit IRC | 19:47 | |
| *** torgomatic has quit IRC | 19:47 | |
| *** _nadya_ has joined #openstack-infra | 19:48 | |
| bnemec | lifeless: Oh, maybe it should have just used diskimage-builder-core directly though. I see that's what t-i-e does. | 19:48 |
| openstackgerrit | A change was merged to openstack-infra/reviewstats: Encode to utf-8 before printing https://review.openstack.org/99104 | 19:49 |
| *** matjazp has quit IRC | 19:49 | |
| clarkb | and 7 is up. now we wait for it to rebalance and when that is done (probably late today I hope). I will disable 01 | 19:49 |
| clarkb | 07 is super write heavy now as it gets shards and the initial sar numbers look good | 19:50 |
| *** torgomatic has joined #openstack-infra | 19:54 | |
| *** ociuhandu has joined #openstack-infra | 19:54 | |
| *** nati_ueno has joined #openstack-infra | 19:54 | |
| jesusaurus | did infra switch over to puppet3? | 19:55 |
| *** Sukhdev has joined #openstack-infra | 19:56 | |
| mriedem | fungi: this was promoted but should i still recheck it? https://review.openstack.org/#/c/99144/ | 19:58 |
| *** ociuhandu has quit IRC | 19:58 | |
| *** zns has joined #openstack-infra | 19:58 | |
| *** adalbas has quit IRC | 19:58 | |
| fungi | mriedem: yeah, it apparently failed i the gate | 19:59 |
| mriedem | ok i rechecked it | 19:59 |
| *** zul has joined #openstack-infra | 19:59 | |
| *** amcrn has joined #openstack-infra | 20:00 | |
| clarkb | jesusaurus: no, but we are trying to be compat with both as we move | 20:00 |
| clarkb | jesusaurus: right now our puppet is compatible with puppet3 on slaves for fedora and trusty | 20:00 |
| nibalizer | clarkb: you scared me with your typo above | 20:01 |
| nibalizer | i'm all 'wth is gpuppet' | 20:01 |
| jesusaurus | clarkb: http://logs.openstack.org/50/99950/1/check/gate-config-puppet-apply-precise/4253068/console.html sure looks like the deprecation warnings are coming from puppet-3.6.2 | 20:01 |
| clarkb | jesusaurus: wow thats new | 20:02 |
| clarkb | jesusaurus: looks like a gemfile installed it? | 20:02 |
| clarkb | nibalizer: the keyboard on my laptop has a bad habit of transposing spaces and letters that end words | 20:02 |
| jesusaurus | i guess... | 20:02 |
| *** james_li has quit IRC | 20:03 | |
| bearhands | clarkb: Supposedly you can try to nuke a lot of those rax instances now, if you've not already... or if we didn't make them just disappear for you | 20:04 |
| clarkb | jesusaurus: that is really odd though. That is why we don't use puppet librarian for example | 20:04 |
| bearhands | their task states may still say deleting, but you can try them again | 20:04 |
| *** maxbit has joined #openstack-infra | 20:04 | |
| clarkb | jesusaurus: I am wondering if something somewhere added a dependency on newer puppet (like lint) | 20:04 |
| clarkb | bearhands: thanks | 20:04 |
| bearhands | np.. let me know if they don't go away :( | 20:04 |
| clarkb | bearhands: according to our big graph a ton of nodes deleted all at once | 20:04 |
| clarkb | so I think it may be sorting itself out | 20:05 |
| bearhands | ok, we may have done it for you | 20:05 |
| bearhands | sounds like we did | 20:05 |
| bearhands | :) | 20:05 |
| bearhands | cools. | 20:05 |
| *** bookwar has joined #openstack-infra | 20:06 | |
| *** _nadya_ has quit IRC | 20:08 | |
| *** todd_dsm has quit IRC | 20:09 | |
| *** blamar has quit IRC | 20:09 | |
| clarkb | mordred: ok do it again | 20:09 |
| *** james_li has joined #openstack-infra | 20:10 | |
| mordred | clarkb: wait - what do I do again? | 20:13 |
| clarkb | @mordred | goooooooooooooooool | 20:15 |
| *** mfer has quit IRC | 20:15 | |
| fungi | bearhands: clarkb: that jump in the graph was me explicitly deleting any nodes nodepool seemed to think it had been trying to delete for at least an hour | 20:18 |
| *** jistr has quit IRC | 20:18 | |
| clarkb | fungi: rgr | 20:18 |
| clarkb | fungi: sdague mordred one other thing we may try with es01 is to turn it into a non data node. Basically point clients at it and only use it for its vast quantity of RAM | 20:19 |
| isviridov | clarkb, mordred, fungi could you give me a hint when it is planned to add new projects to stackforge? Last time it was on Fridays | 20:19 |
| clarkb | fungi: sdague mordred but I don't think we need to do that | 20:19 |
| fungi | all of the instances in the list for my batch delete were successfully removed, which does confirm the error nodes were either fixed or gone | 20:19 |
| clarkb | isviridov: we do it as we are able to review the changes now. unfortunately I haven't been able to do much review this week | 20:19 |
| *** todd_dsm has joined #openstack-infra | 20:19 | |
| fungi | few if any of us have been able to get any reviewing done for a couple of weeks | 20:20 |
| *** matjazp has joined #openstack-infra | 20:20 | |
| fungi | except for reviewing changes fixing breakage i the gate or other parts of our infrastructure | 20:20 |
| clarkb | mordred: I hope you are watching | 20:21 |
| adam_g | mordred, https://review.openstack.org/#/c/99740/ | 20:21 |
| jesusaurus | clarkb: aha! install_puppet.sh is only pinning the version on trusty | 20:23 |
| jesusaurus | clarkb: not on precise | 20:23 |
| *** rcarrill` has joined #openstack-infra | 20:24 | |
| mordred | clarkb: gooooooooooooooool | 20:24 |
| mordred | :) | 20:24 |
| clarkb | jesusaurus: thats uh unfortunate. But maybe if it works... | 20:25 |
| clarkb | I guess it makes getting that tset voting harder? | 20:25 |
| *** matjazp has quit IRC | 20:25 | |
| *** miqui has quit IRC | 20:25 | |
| clarkb | jesusaurus: I am actually somewhat curious as to whether people think this may be perferred or not | 20:25 |
| openstackgerrit | K Jonathan Harker proposed a change to openstack-infra/config: Pin the version of puppet on all Ubuntus https://review.openstack.org/100010 | 20:25 |
| mordred | clarkb, jesusaurus: we _Wanted_ to pin on precise ... | 20:25 |
| clarkb | I can go either way right now | 20:25 |
| *** rcarrillocruz has quit IRC | 20:25 | |
| clarkb | mordred: so I wonder if I have puppet 3 on es07 | 20:26 |
| mordred | clarkb: wait - you were the one who convinced me to pin everywhere | 20:26 |
| clarkb | mordred: also this drama is great | 20:26 |
| clarkb | mordred: yes still pin | 20:26 |
| clarkb | mordred: but bump slave pin to 3 if it iworks | 20:26 |
| *** todd_dsm has quit IRC | 20:26 | |
| clarkb | mordred: but then we would have two different versions of puppet on precise whcih isn't a good thing | 20:26 |
| clarkb | so I can go either way and going back to what we had is probably simplest right now | 20:27 |
| mordred | so - somewhere I thought we had a patch to install_puppet that made all of this correct | 20:27 |
| clarkb | ok puppet on es07 is correct | 20:27 |
| mordred | we need to pin to 2.x on != trusty - and to 3.x on trusty | 20:27 |
| jesusaurus | pin to 3 on trusty? | 20:28 |
| clarkb | right | 20:28 |
| mordred | jesusaurus: yeah - clarkb thinks we should be explicit with our pins everywhere | 20:28 |
| clarkb | jesusaurus: however I just booted a node and it worked fine | 20:28 |
| clarkb | jesusaurus: so I don't know how this is breaking on the slaves | 20:28 |
| mordred | clarkb: we should take a day and just migrated | 20:28 |
| mordred | so we can stop screwing with this dance | 20:29 |
| clarkb | ++ | 20:29 |
| clarkb | mordred: also O_O are you watching this game? | 20:29 |
| mordred | gooooooooooooooooooooooooooooooooooooooooooooooool | 20:29 |
| clarkb | jesusaurus: how is it installing 3 on precise? | 20:30 |
| clarkb | it clearly is, but I still don't understand how | 20:30 |
| mordred | clarkb: is it because we're doing gems for the puppet tests with rvm or something? | 20:30 |
| *** alexpilotti has quit IRC | 20:30 | |
| clarkb | mordred: thats what I am thinking | 20:30 |
| *** zzelle has quit IRC | 20:30 | |
| mordred | yeah. it's time to be done with this game | 20:30 |
| mordred | wait | 20:30 |
| mordred | not the game | 20:30 |
| mordred | the game is great | 20:31 |
| mordred | the puppet2/3 split | 20:31 |
| mordred | it officially bores me now | 20:31 |
| *** zzelle has joined #openstack-infra | 20:31 | |
| jesusaurus | clarkb: yeah, i dont understand how either. i had misread "!= trusty" as "== trusty" in install_puppet.sh | 20:32 |
| openstackgerrit | Jaroslav Henner proposed a change to openstack-infra/jenkins-job-builder: add presend-script to email-ext https://review.openstack.org/99994 | 20:32 |
| *** sandywalsh_ has quit IRC | 20:33 | |
| *** flaper87|afk is now known as flaper87 | 20:33 | |
| *** mwagner_lap has quit IRC | 20:36 | |
| mordred | gooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooool | 20:37 |
| clarkb | mordred: this ame | 20:37 |
| mordred | clarkb: dude | 20:37 |
| clarkb | I approve | 20:37 |
| *** eharney has quit IRC | 20:38 | |
| krotscheck | mordred: You’ve seen this? http://fivethirtyeight.com/interactives/world-cup/ | 20:40 |
| krotscheck | Spain was heavily favored over the netherlands :) | 20:40 |
| mordred | krotscheck: I had not seen that - but I believe that they were wrong :) | 20:41 |
| mordred | wow, and they thought Chile over Holland too? | 20:41 |
| mordred | (I hope the other bits of that are right though - if so, it would mean I'd get to see England v. Columbia, which would be a cool match to watch) | 20:42 |
| clarkb | mordred: this game | 20:44 |
| clarkb | it doesn't stop | 20:44 |
| mordred | dude | 20:44 |
| mordred | I think they're potentially still pissed off about last time | 20:44 |
| jogo | mordred: what happened to you going to the world cup? | 20:46 |
| *** dkliban is now known as dkliban_afk | 20:48 | |
| mordred | jogo: I'll be going in a couple of weeks | 20:48 |
| mordred | I fly down on the 25th, then watch a game in rio on the 28th (winner of C vs. runner up of D) - and spend the rest of the time watching matches in bars with brazilian friends | 20:49 |
| clarkb | mordred: when do you go to germany? | 20:49 |
| *** markwash has joined #openstack-infra | 20:52 | |
| jogo | mordred: oh nice | 20:53 |
| *** Ryan_Lane1 has joined #openstack-infra | 20:55 | |
| SpamapS | clarkb: whenever he damn well pleases? ;) | 20:56 |
| SpamapS | jogo: so if I wanted to use 1 of my hours today to help improve the gate.. where would I start? | 20:56 |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Revert "Set CEILOMETER_PIPELINE_INTERVAL to 15" https://review.openstack.org/98871 | 20:56 |
| lifeless | mordred: oh so we do crossover in sunnyvale? | 20:57 |
| *** rfolco has quit IRC | 20:57 | |
| SpamapS | aww man I was going to grab https://bugs.launchpad.net/devstack/+bug/1253482 | 20:58 |
| uvirtbot | Launchpad bug 1253482 in devstack "Keystone's IANA-assigned default port in linux local ephemeral port range" [Undecided,In progress] | 20:58 |
| jogo | SpamapS: excellent question | 20:59 |
| jogo | SpamapS: fix the heat bugs listed on http://status.openstack.org/elastic-recheck/index.html | 21:00 |
| *** weshay has quit IRC | 21:01 | |
| *** radez is now known as radez_g0n3 | 21:01 | |
| *** james_li has quit IRC | 21:01 | |
| SpamapS | ah silly me I was looking at /rechecks | 21:01 |
| jogo | SpamapS: that or work on helping us drop gate jobs | 21:02 |
| mordred | lifeless: nope. I'll be in NYC before I fly out | 21:02 |
| mordred | clarkb: on the saturday - so I'll be in germany for the game. | 21:02 |
| jogo | as per the 'test nodes' graph at the bottom of http://status.openstack.org/zuul/ | 21:02 |
| jogo | we are hitting quota | 21:02 |
| jogo | SpamapS: also it looks like are having a spike in grenade issue, I think mriedem was looking into those | 21:04 |
| clarkb | mordred: woot | 21:05 |
| *** sarob_ has joined #openstack-infra | 21:05 | |
| *** ihrachyshka has quit IRC | 21:05 | |
| *** james_li has joined #openstack-infra | 21:10 | |
| *** shayneburgess has quit IRC | 21:11 | |
| *** mgagne has quit IRC | 21:11 | |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Add unit tests for SearchEngine.search https://review.openstack.org/100017 | 21:13 |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Add unit tests for SearchEngine.search https://review.openstack.org/100017 | 21:13 |
| *** cp16net_ has quit IRC | 21:13 | |
| *** weshay has joined #openstack-infra | 21:14 | |
| *** sandywalsh_ has joined #openstack-infra | 21:14 | |
| *** sandywalsh_ has quit IRC | 21:14 | |
| *** otherwiseguy has quit IRC | 21:18 | |
| *** smarcet has quit IRC | 21:19 | |
| *** mgagne has joined #openstack-infra | 21:20 | |
| *** mgagne is now known as Guest61486 | 21:20 | |
| *** matjazp has joined #openstack-infra | 21:21 | |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/jenkins-job-builder: Apply defaults to job-templates parameters https://review.openstack.org/100020 | 21:23 |
| *** hashar has joined #openstack-infra | 21:24 | |
| *** matjazp has quit IRC | 21:25 | |
| *** todd_dsm has joined #openstack-infra | 21:27 | |
| *** wenlock_ has quit IRC | 21:28 | |
| jogo | sdague: wow a single devstack-gate patch takes up 20+ jobs | 21:28 |
| *** rcarrillocruz has joined #openstack-infra | 21:30 | |
| yjiang5 | clarkb: hi, all, I'm reading http://docs.openstack.org/developer/tempest/HACKING.html#parallel-test-execution and I didn't find any lock in AggregatesAdminTest which is specified in the doc, is it because doc not updated for code change? | 21:31 |
| *** rcarrill` has quit IRC | 21:31 | |
| clarkb | yjiang5: I am not sure. mtreinish or the folks in the qa channel may know | 21:31 |
| mordred | jogo: yah man. | 21:31 |
| clarkb | yjiang5: but if I had to guess I would guess they were able to make it lockless and the docs aren't up to date | 21:32 |
| mtreinish | yjiang5: locks are only needed on the aggregates tests if there is host manipulation in the test | 21:33 |
| *** isviridov is now known as isviridov|away | 21:34 | |
| mtreinish | yjiang5: for example: http://git.openstack.org/cgit/openstack/tempest/tree/tempest/api/compute/admin/test_aggregates.py#n154 | 21:35 |
| *** e0ne has quit IRC | 21:38 | |
| *** sarob_ has quit IRC | 21:38 | |
| *** e0ne has joined #openstack-infra | 21:39 | |
| *** mrodden has quit IRC | 21:39 | |
| *** sarob_ has joined #openstack-infra | 21:39 | |
| mtreinish | jogo: I think tempest wins at 26 | 21:41 |
| *** maxbit has quit IRC | 21:41 | |
| jogo | mtreinish: wow | 21:42 |
| jogo | mtreinish: so stop working on tempest your hogging resources ;) | 21:42 |
| *** e0ne has quit IRC | 21:42 | |
| *** sarob_ has quit IRC | 21:43 | |
| mtreinish | jogo: nah, I think I should push more hacking patches. I skipped 2 rules because there were too many hits on them | 21:43 |
| *** shayneburgess has joined #openstack-infra | 21:44 | |
| yjiang5 | mtreinish: thanks. I searched 'lock', should use "Lock". Do you mind me to send a patch to update the doc to point to the function? | 21:47 |
| yjiang5 | clarkb: thanks. | 21:47 |
| *** fbo_away is now known as fbo | 21:48 | |
| *** todd_dsm has quit IRC | 21:49 | |
| jogo | mtreinish: which ones? | 21:49 |
| mtreinish | yjiang5: go for it, but I'm not sure how specific we should get in the docs because how we instantiate a lock has changed at least 3 times | 21:49 |
| mtreinish | jogo: H405 and H904 | 21:50 |
| yjiang5 | mtreinish: aha, got it. That make sense also. Hope others can find the usage more easily. | 21:50 |
| jogo | ahh yeah, 904 is a little intense | 21:51 |
| jogo | and 405 can be really big to fix as well. nova skips a bunch for similar reasons | 21:51 |
| mtreinish | jogo: yeah and we were already skipping h404 before the new version because docstrings in tempest suck | 21:52 |
| mtreinish | so h405 will take some time | 21:52 |
| *** markwash has quit IRC | 21:56 | |
| jogo | for sure | 21:56 |
| *** Sukhdev has quit IRC | 21:56 | |
| *** hashar has quit IRC | 21:57 | |
| *** e0ne has joined #openstack-infra | 21:59 | |
| devananda | sdague: fwiw, it seems like ironic does indeed have a separate merge queue now... I'm not sure what changed to effect that, though | 21:59 |
| devananda | clarkb: ^ ? | 21:59 |
| *** xyang1 has quit IRC | 21:59 | |
| clarkb | devananda: sdague wrote a chnage to stop gatin gon the oslo stuff | 21:59 |
| devananda | clarkb: ahh! great | 21:59 |
| clarkb | devananda: which decoupled the transitive test dependency | 21:59 |
| devananda | fantastic | 21:59 |
| jogo | clarkb: oh nice | 22:00 |
| jogo | so random question: so we have non-voting staable icehouse and havana jobs for all the clients | 22:00 |
| jogo | as client aren't tied to a release | 22:00 |
| clarkb | right | 22:00 |
| jogo | and we have neutron and nova-network tests for both | 22:01 |
| jogo | perhaps we can strip drop the number of tests there | 22:01 |
| clarkb | possibly | 22:02 |
| clarkb | does neutronclient run the nova network tests? | 22:02 |
| clarkb | it probably shouldn't but we need to test novaclient with neutronclient in the neutron case | 22:02 |
| jogo | maybe say just nova-network for all clients except neutron which would have neutron client? | 22:03 |
| jogo | so drop 2 jobs for most client patches | 22:03 |
| jogo | in check queue | 22:03 |
| *** harlowja has quit IRC | 22:03 | |
| jogo | clarkb: not sure what you mean | 22:04 |
| *** aysyd has quit IRC | 22:04 | |
| *** timrc-afk is now known as timrc | 22:04 | |
| jogo | neutronclient isn't used in the nova-network settings | 22:04 |
| jogo | and if it is at all that is a bug | 22:04 |
| jogo | in nova | 22:04 |
| clarkb | jogo: I am saying novaclient needs to be tested on neutron too | 22:05 |
| jogo | but don't think it is | 22:05 |
| clarkb | right that is the case aiui. its the neutron case that is tricky | 22:05 |
| jogo | clarkb: ohh hmm yeah there are some nova comands that talk to neutron | 22:05 |
| jogo | in the backend | 22:05 |
| jogo | except our tempest tests don't do much around testing the clients directly | 22:06 |
| jogo | this is much more for the interaction between services (which uses the clients) | 22:06 |
| clarkb | right | 22:06 |
| clarkb | the whole point of those tests is that services depend on the clients but we don't really gate them against old stable versions | 22:06 |
| clarkb | so we have to be careful that we cover the intended bases | 22:07 |
| jogo | clarkb: yeah we may loose some coverage by dropping some of those jobs | 22:08 |
| jogo | dropping cells test would help too ;) | 22:08 |
| jogo | its pretty useless http://logs.openstack.org/51/99751/2/check/check-devstack-dsvm-cells/9380c31/console.html#_2014-06-13_20_40_23_139 | 22:08 |
| jogo | it runs only a few tests and most aren't cells related | 22:09 |
| jogo | but thats a whole can of worms | 22:09 |
| *** sarob_ has joined #openstack-infra | 22:10 | |
| *** blamar has joined #openstack-infra | 22:10 | |
| *** marcoemorais has quit IRC | 22:10 | |
| *** marcoemorais has joined #openstack-infra | 22:11 | |
| *** sarob__ has joined #openstack-infra | 22:11 | |
| *** flaper87 is now known as flaper87|afk | 22:13 | |
| *** sarob___ has joined #openstack-infra | 22:13 | |
| *** sarob_ has quit IRC | 22:14 | |
| *** sarob__ has quit IRC | 22:16 | |
| jogo | clarkb: anyone put up the patch to drop some postgres jobs yet? | 22:17 |
| clarkb | jogo: I don't think so. What are we trying to accomplish with that? | 22:17 |
| clarkb | postgres actually isn't very problematic... | 22:17 |
| jogo | clarkb: free capacity. actually you have been very quite on that ML thread | 22:17 |
| *** james_li has quit IRC | 22:17 | |
| jogo | [openstack-dev] Gate proposal - drop Postgresql configurations in the gate | 22:18 |
| clarkb | jogo: I was apparently getting a lot of bounces | 22:18 |
| clarkb | so mailman made me go away | 22:18 |
| *** sarob___ has quit IRC | 22:18 | |
| jogo | ahh, well yeah that thread is why | 22:18 |
| jogo | so the idea I think we are moving towards is leaving one job in postgres mode | 22:18 |
| jogo | maybe one of the neutron jobs | 22:18 |
| clarkb | ya I was beginning to wonder why my email was so quiet then eventually mailman told me | 22:19 |
| clarkb | I recently resubscribed | 22:19 |
| jogo | haha | 22:19 |
| anteaya | jhesketh: mtreinish submitted a talk to the openstack miniconf for pycon au | 22:20 |
| clarkb | yup never got that thread | 22:20 |
| anteaya | it is going to be a great talk, the crowds will just pack themselves in | 22:20 |
| anteaya | you have to accept it | 22:20 |
| jogo | clarkb: http://lists.openstack.org/pipermail/openstack-dev/2014-June/037431.html | 22:20 |
| jogo | clarkb: happy reading, I am very interested in your thoughts on it | 22:21 |
| *** prad has quit IRC | 22:21 | |
| *** matjazp has joined #openstack-infra | 22:22 | |
| mtreinish | anteaya: I'm not sure that I can live up to that :) | 22:22 |
| anteaya | you did already | 22:22 |
| anteaya | the room was packed | 22:22 |
| anteaya | they gave you an encore | 22:22 |
| anteaya | it was inspiriing | 22:22 |
| anteaya | :D | 22:22 |
| jogo | anteaya: what was the talk? | 22:22 |
| *** matjazp_ has joined #openstack-infra | 22:23 | |
| anteaya | mtreinish's tempest talk at summit | 22:23 |
| anteaya | you didn't hear about it at the time | 22:23 |
| anteaya | oh it was marvelous | 22:23 |
| anteaya | what 350 people in that room? | 22:23 |
| anteaya | they were asking for autographs afterward | 22:23 |
| anteaya | :D | 22:23 |
| *** blamar has quit IRC | 22:24 | |
| mtreinish | hahaha, I guess the room was mostly full. But I can safetly say that I didn't use a pen except to sign reciepts my whole stay in atlanta... | 22:25 |
| anteaya | shhhhhhh | 22:26 |
| *** mriedem has quit IRC | 22:26 | |
| anteaya | :D | 22:26 |
| *** matjazp has quit IRC | 22:27 | |
| jroll | is there a way for a patch to depend on multiple (chains of) patches, without rebasing everything into a massive chain? | 22:28 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/config: Remove duplicate dsvm-postgres-full jobs https://review.openstack.org/100032 | 22:28 |
| clarkb | jogo: ugh so I think there is a clear issue in people not groking the question because it probably could have been posed more clearly | 22:29 |
| clarkb | jogo: no one seems to be saying stop testing postgres | 22:29 |
| openstackgerrit | Salvatore Orlando proposed a change to openstack-infra/config: Make neutron full job voting (neutron gate only) https://review.openstack.org/88289 | 22:29 |
| clarkb | jogo: it seems like sdague is saying stop making postgres a special job | 22:29 |
| clarkb | jogo: and we can test in unittests all we want | 22:29 |
| *** changbl has quit IRC | 22:29 | |
| jogo | clarkb: exactly | 22:29 |
| jogo | so this is what I am thinking we should do: patch coming soon | 22:30 |
| *** zns has quit IRC | 22:30 | |
| *** CaptTofu_ has quit IRC | 22:30 | |
| fungi | yeah, other participants in the thread not getting that the bits of postgres actually exercised differently than mysql behind a tempest job are likely trivial to nonexistent | 22:30 |
| mtreinish | jogo: did you see my proposal on the thread? | 22:31 |
| mtreinish | to switch the global neutron job to postgres and run the mysql neutron only on neutron patches | 22:31 |
| clarkb | jogo: well I think we need to reset the whole damn thread | 22:32 |
| clarkb | and start with a concrete proposal. | 22:32 |
| clarkb | drop the special postgres job. run postgres as DB backend for another job. Stress that postgres isn't going anywhere for unittest nodes | 22:32 |
| *** sarob_ has joined #openstack-infra | 22:33 | |
| clarkb | I feel like everyone was missing the very important point that unittests/functional tests can use postgres all they want | 22:33 |
| *** UtahDave has quit IRC | 22:33 | |
| clarkb | eg DB migrations are tested against postgres and mysql assuming oslo db is working | 22:33 |
| *** matjazp_ has quit IRC | 22:34 | |
| *** zns has joined #openstack-infra | 22:34 | |
| *** blamar has joined #openstack-infra | 22:34 | |
| fungi | of everyone who has claimed in that thread that testing with postgres turned up postgres-specific issues, nobody has actually come forward suggesting they were discovered because of fails on tempest tests | 22:34 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/config: Merge postgres and neutron jobs in integrated-gate template https://review.openstack.org/100033 | 22:34 |
| jogo | clarkb mtreinish: this is my proposal, as a starting point | 22:34 |
| jogo | mtreinish: like this ^ | 22:34 |
| jogo | fungi: ^ | 22:35 |
| *** dangers is now known as dangers_away | 22:35 | |
| clarkb | fungi: correct, they were all catchable functional/unittest type things | 22:35 |
| *** sarob__ has joined #openstack-infra | 22:35 | |
| clarkb | honestly though postgres seems like a weird target because it hasn't really been a problem | 22:36 |
| mtreinish | jogo: that's basically what I was saying, but think once we get to tempest-neutron-full working you shouldn't need the extra nova job | 22:36 |
| jogo | mtreinish: I added that just for good meaure | 22:36 |
| clarkb | mtreinish: right | 22:36 |
| clarkb | mtreinish: that was sort of the basis of my test matrices sessions | 22:36 |
| jogo | mtreinish: but happy to drop it if you think that is the right direction | 22:36 |
| jogo | clarkb: yeah | 22:36 |
| clarkb | once you have feature parity this is much easier | 22:36 |
| mtreinish | clarkb: I agree we don't really need the extra config, but I was just suggesting compacting things to appease the greater crowd | 22:37 |
| anteaya | jroll: multiple chains of patches? do you mean across projects? | 22:37 |
| mtreinish | jogo: yeah short time it's not the same thing | 22:37 |
| clarkb | mtreinish: ya I think its a reasonable compromise | 22:37 |
| mtreinish | but once the last set of neutron bugs get fixed we can drop the extra nova job | 22:37 |
| clarkb | jroll: the answer whether a single project or multiple projects is "no" | 22:38 |
| mtreinish | although I think the extra coverage is minimal | 22:38 |
| clarkb | anteaya: ^ | 22:38 |
| jroll | anteaya: no. in my case, my patch depends on three in-flight patches | 22:38 |
| clarkb | jroll: anteaya the exception is you can push a merge commit, but we turn those off | 22:38 |
| *** sarob_ has quit IRC | 22:38 | |
| jroll | clarkb: :( | 22:38 |
| *** Guest61486 has quit IRC | 22:38 | |
| jogo | mtreinish: so your saying drop the nova change in https://review.openstack.org/#/c/100033/1/modules/openstack_project/files/zuul/layout.yaml ? | 22:38 |
| clarkb | jroll: you can have a soft dependency and construct it such that tests don't pass without all three and let reviewers know about the dependency | 22:38 |
| *** zns has quit IRC | 22:39 | |
| jroll | clarkb: any idea what people do in this situation? (the chains are owned by different people, I'm not going to rebase their patches) | 22:39 |
| jroll | ah | 22:39 |
| clarkb | but usually you rebase | 22:39 |
| clarkb | jroll: really people shouldn't be afraid of having their code manipulated for the greater good | 22:39 |
| * fungi is not afraid to rebase other devs' patches | 22:39 | |
| mtreinish | jogo: yeah, I don't think the extra tests run there will provide much extra coverage | 22:39 |
| fungi | (or fix obvious trivial erors in them) | 22:39 |
| fungi | s/erors/errors/ | 22:39 |
| mtreinish | jogo: although I'm forgetting about nova-net | 22:39 |
| jroll | clarkb: hmm, I'd still want to poke them first, idk. I'll figure something out, thanks. | 22:40 |
| jogo | mtreinish: yeah see the commit message ;) | 22:40 |
| clarkb | jroll: really meh | 22:40 |
| jogo | postgres+nova-network will be tested in nova. | 22:40 |
| clarkb | jroll: I think this is a social thing we may need to push harder to change | 22:40 |
| mtreinish | jogo: heh, yeah I should start reading those | 22:40 |
| clarkb | jroll: it shouldn't be seen as offensive or a problem for people to do stuff like this | 22:40 |
| jogo | mtreinish: I am gonna float that patch on the thread so we can start discussing something concrete | 22:41 |
| *** andreykurilin_ has quit IRC | 22:41 | |
| *** mgagne has joined #openstack-infra | 22:41 | |
| *** mwagner_lap has joined #openstack-infra | 22:41 | |
| *** mgagne is now known as Guest8031 | 22:41 | |
| jroll | clarkb: I agree, I just don't like pissing off people who may disagree :) | 22:41 |
| jroll | clarkb: my main thing is, the dependent patches are completely unrelated, other than the fact I depend on them, so I don't want one to get blocked on the other landing | 22:42 |
| JayF | FWIW, anyone feel free to rebase my patches or fix bugs in them whenever you want :) | 22:42 |
| JayF | except you, jroll | 22:43 |
| jroll | -.- | 22:43 |
| fungi | territoriality over patches is something we should definitely strive to rise above as a community | 22:44 |
| clarkb | jroll: so that sort of situation is an outlier and the sfot dependency approach may be best if they are not actually related | 22:44 |
| fungi | there's already PLENTY of work to go around without anyone needing to carve out their own exclusive problem space | 22:44 |
| jroll | indeed | 22:44 |
| *** CaptTofu_ has joined #openstack-infra | 22:45 | |
| jroll | I'll look a bit more at it | 22:45 |
| jroll | one is about to land; one is far from landing | 22:45 |
| jroll | so it might work out if I order correctly | 22:45 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/config: Merge postgres and neutron jobs in integrated-gate template https://review.openstack.org/100033 | 22:46 |
| *** amcrn has quit IRC | 22:47 | |
| openstackgerrit | A change was merged to openstack/requirements: Fix sphinx requirement to add overlap https://review.openstack.org/98947 | 22:49 |
| *** zns has joined #openstack-infra | 22:49 | |
| openstackgerrit | OpenStack Proposal Bot proposed a change to openstack-dev/hacking: Updated from global requirements https://review.openstack.org/100037 | 22:50 |
| *** blamar has quit IRC | 22:55 | |
| *** ramashri has quit IRC | 22:59 | |
| *** zns has quit IRC | 23:02 | |
| *** markmcclain has quit IRC | 23:03 | |
| *** fbo is now known as fbo_away | 23:07 | |
| *** matjazp has joined #openstack-infra | 23:08 | |
| *** wenlock_ has joined #openstack-infra | 23:10 | |
| *** matjazp has quit IRC | 23:12 | |
| *** reed has quit IRC | 23:15 | |
| *** medieval1 has quit IRC | 23:17 | |
| *** medieval1 has joined #openstack-infra | 23:17 | |
| *** medieval1 has quit IRC | 23:22 | |
| SpamapS | jogo: what is "integrated-gate-neutron" ? | 23:23 |
| mtreinish | SpamapS: it's a job template in zuul that gets run on neutron and neutronclient change | 23:26 |
| *** e0ne has quit IRC | 23:28 | |
| *** e0ne has joined #openstack-infra | 23:29 | |
| morganfainberg | mtreinish, so this means instead of doing the dsvm-neutron-full we do dsvm-neutron-pg, if i'm reading this correctly; postgres-ful is removed from the main python27 templates and added specifically to nova? | 23:32 |
| *** melwitt has quit IRC | 23:35 | |
| mtreinish | morganfainberg: actually it replaces dsvm-neutron in the integrated-gate template. The neutron-full job is a non-voting job because neutron still isn't stable enough to run all the tests in parallel | 23:35 |
| mtreinish | but yeah that's the basic idea | 23:35 |
| morganfainberg | mtreinish, ah | 23:35 |
| morganfainberg | mtreinish, ok. and then the postgres+novanet is covered on nova changes only | 23:36 |
| * SpamapS mumbles something about that PERHAPS being the actual problem | 23:36 | |
| mtreinish | yep | 23:36 |
| *** lcostantino has joined #openstack-infra | 23:36 | |
| SpamapS | well time for the weekend to start | 23:36 |
| morganfainberg | SpamapS, ahve a good weekend :) | 23:37 |
| morganfainberg | mtreinish, ok. i can be on board with that. | 23:38 |
| *** matjazp has joined #openstack-infra | 23:38 | |
| mtreinish | morganfainberg: yeah it seems like the best compromise | 23:40 |
| *** matjazp_ has joined #openstack-infra | 23:40 | |
| *** shayneburgess has quit IRC | 23:40 | |
| openstackgerrit | Morgan Fainberg proposed a change to openstack-infra/config: Make apache-services tempest check experimental https://review.openstack.org/97638 | 23:41 |
| *** matjazp has quit IRC | 23:43 | |
| *** matjazp_ has quit IRC | 23:44 | |
| *** dims has quit IRC | 23:47 | |
| *** praneshp has quit IRC | 23:53 | |
| *** sweston has quit IRC | 23:57 | |
| *** medieval1 has joined #openstack-infra | 23:58 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!