| *** otter768 has quit IRC | 00:01 | |
| *** otter768 has joined #openstack-infra | 00:01 | |
| *** heyongli has quit IRC | 00:03 | |
| *** garyk has quit IRC | 00:03 | |
| *** MarkAtwood has joined #openstack-infra | 00:03 | |
| *** ybathia has quit IRC | 00:03 | |
| *** heyongli has joined #openstack-infra | 00:04 | |
| *** garyk has joined #openstack-infra | 00:04 | |
| *** eharney has quit IRC | 00:05 | |
| *** yamamoto has joined #openstack-infra | 00:06 | |
| ruagair | I might just remove it and let chaos reign. | 00:06 |
|---|---|---|
| *** btully has quit IRC | 00:07 | |
| mriedem | is logstash.o.o down? | 00:08 |
| *** geoffarnold has quit IRC | 00:08 | |
| *** geoffarn_ has joined #openstack-infra | 00:08 | |
| mordred | ruagair: chaos \o/ | 00:09 |
| clarkb | mriedem: I don't think so? | 00:09 |
| mriedem | haven't been able to get anything back in awhile | 00:10 |
| mriedem | for | 00:10 |
| mriedem | message:"in test_exceptions_raise" AND tags:"console" | 00:10 |
| *** achanda has joined #openstack-infra | 00:11 | |
| *** salv-orl_ has quit IRC | 00:11 | |
| clarkb | mriedem: are you doing an all time serach? | 00:11 |
| *** otter768 has quit IRC | 00:12 | |
| *** otter768 has joined #openstack-infra | 00:12 | |
| krotscheck | nibalizer: I am writing ruby. I blame you for the deep scourging wounds across my soul. | 00:13 |
| *** heyongli has quit IRC | 00:13 | |
| *** achanda has quit IRC | 00:13 | |
| *** mestery has joined #openstack-infra | 00:14 | |
| *** heyongli has joined #openstack-infra | 00:14 | |
| *** hdd has joined #openstack-infra | 00:18 | |
| *** fbo has quit IRC | 00:18 | |
| *** mestery has quit IRC | 00:19 | |
| mriedem | clarkb: 12 hours | 00:19 |
| *** juggler_ has quit IRC | 00:19 | |
| *** fbo has joined #openstack-infra | 00:22 | |
| *** sputnik13 has quit IRC | 00:22 | |
| *** garyk has quit IRC | 00:23 | |
| *** heyongli has quit IRC | 00:23 | |
| *** heyongli has joined #openstack-infra | 00:24 | |
| *** stevemar_ has quit IRC | 00:25 | |
| clarkb | mriedem: did it return a 500 or never return? | 00:28 |
| clarkb | mriedem: it returns a 500 for me | 00:28 |
| *** DericHorn-HP has joined #openstack-infra | 00:28 | |
| *** geoffarn_ has quit IRC | 00:28 | |
| *** geoffarnold has joined #openstack-infra | 00:29 | |
| *** stevemar_ has joined #openstack-infra | 00:29 | |
| *** gyee has quit IRC | 00:31 | |
| *** otter768 has quit IRC | 00:32 | |
| *** otter768 has joined #openstack-infra | 00:32 | |
| mriedem | i was getting a 502 | 00:32 |
| mriedem | now 500 | 00:32 |
| *** skylerberg has quit IRC | 00:33 | |
| *** heyongli has quit IRC | 00:34 | |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config: (Re)add midonet tempest jobs https://review.openstack.org/230932 | 00:34 |
| openstackgerrit | Tony Breeds proposed openstack/requirements: Add virtualenv to global requirements https://review.openstack.org/227239 | 00:35 |
| *** heyongli has joined #openstack-infra | 00:35 | |
| *** vmbrasseur is now known as vmb_out | 00:35 | |
| clarkb | mriedem: the cluster doesn't seem particularly unhappy let me look at logs | 00:38 |
| clarkb | mriedem: looks like kibana may be the thing failing I will attempt to restart it | 00:39 |
| openstackgerrit | Matt Riedemann proposed openstack/requirements: Fix cap on python-glanceclient https://review.openstack.org/233872 | 00:39 |
| clarkb | the search fail doesn't show up in es logs | 00:39 |
| mriedem | yeah es seems fine | 00:41 |
| mriedem | err e-r i mean | 00:41 |
| *** pvaneck has quit IRC | 00:42 | |
| *** angdraug has quit IRC | 00:43 | |
| *** heyongli has quit IRC | 00:44 | |
| *** heyongli has joined #openstack-infra | 00:45 | |
| *** asettle has quit IRC | 00:46 | |
| clarkb | mriedem: well that seems ot have creashed firefox | 00:47 |
| mriedem | ha | 00:48 |
| openstackgerrit | Merged openstack/requirements: Temporarily cap webob<1.5.0 in g-r to unblock nova https://review.openstack.org/233851 | 00:48 |
| *** tiswanso_ has quit IRC | 00:48 | |
| *** tiswanso has joined #openstack-infra | 00:49 | |
| *** thorst has joined #openstack-infra | 00:49 | |
| *** geoffarnold has quit IRC | 00:50 | |
| *** geoffarnold has joined #openstack-infra | 00:50 | |
| clarkb | mriedem: any different for you? I did restart kibana | 00:50 |
| *** alexsyip has quit IRC | 00:50 | |
| mriedem | getting 502s | 00:50 |
| clarkb | mriedem: I need to drop off and family shortly (dinner!) but if you want a work around you can grab the kibana 3 source and its js and run that directly against the es api endpoint | 00:51 |
| clarkb | mriedem: sorry I can't get the hosted service working right now | 00:51 |
| mriedem | ok, np | 00:51 |
| * clarkb moves hosting kibana3 up the list | 00:52 | |
| *** lykinsbd_ has joined #openstack-infra | 00:52 | |
| *** lykinsbd has quit IRC | 00:53 | |
| *** gildub has quit IRC | 00:53 | |
| anteaya | carl_baldwin: sorry I wasn't here earlier, I have been eating, hope you got what you needed to move forward | 00:53 |
| * anteaya is glad thanksgiving only happens once a year | 00:53 | |
| anteaya | oh and turnips and beets with butter and coriander | 00:53 |
| clarkb | anteaya: I know you don't turkey but do meal items up north generally mirror those down south? | 00:54 |
| *** asettle has joined #openstack-infra | 00:54 | |
| anteaya | yes | 00:54 |
| *** heyongli has quit IRC | 00:54 | |
| anteaya | potatoes, carrots, stuffing, beets, cauliflour, turnips, squash, sweet potatoes | 00:55 |
| anteaya | and if you can make it pumpkin pie | 00:55 |
| anteaya | I couldn't make it to pie tonight | 00:55 |
| clarkb | haha | 00:55 |
| anteaya | I kept tripping on the turnips and beets | 00:55 |
| *** otter768 has quit IRC | 00:55 | |
| anteaya | it was amazing | 00:55 |
| *** heyongli has joined #openstack-infra | 00:55 | |
| *** thorst has quit IRC | 00:55 | |
| *** otter768 has joined #openstack-infra | 00:55 | |
| anteaya | oh and green beans | 00:55 |
| clarkb | I am debating trying to make smoked turkey for the first time | 00:55 |
| anteaya | and cranberries | 00:55 |
| *** thorst has joined #openstack-infra | 00:56 | |
| anteaya | interesting | 00:56 |
| anteaya | worth a try | 00:56 |
| anteaya | I would suggest just a breast | 00:56 |
| clarkb | would have to get up even earlire than normal but I bet its worth it | 00:56 |
| anteaya | it would be worth the experiment | 00:56 |
| anteaya | and have a back up plan | 00:56 |
| anteaya | if you are having a big crowd | 00:56 |
| anteaya | american thanksgiving is one day, the thursday | 00:56 |
| clarkb | anteaya: ya usually the giant extended family gets together so >1 turkey and meat option will be presnet | 00:56 |
| anteaya | canadian thanksgiving goes all weekend | 00:57 |
| anteaya | cool | 00:57 |
| clarkb | hrm clearly we fail at this thanksgiving thing | 00:57 |
| anteaya | this is the third night of eating in a row | 00:57 |
| anteaya | no, I wouldn't say that at all | 00:57 |
| anteaya | the food sales going into it are amazing | 00:57 |
| anteaya | and the one day of eating is pretty impressive | 00:57 |
| *** hdd has quit IRC | 00:57 | |
| anteaya | my father smoked a lot of things | 00:58 |
| *** Swami has quit IRC | 00:58 | |
| anteaya | best to keep the thin edges to a minimum | 00:58 |
| anteaya | big chunk of meat, smooth on the sides | 00:58 |
| anteaya | thin edges are just inedible | 00:58 |
| anteaya | my father smoked a lot of food | 00:59 |
| clarkb | anteaya: ya but I could eat like that all weekend? clearly we have room to improve | 00:59 |
| anteaya | I just re-read what I wrote | 00:59 |
| anteaya | ha ha ha | 00:59 |
| anteaya | you can start a national campaign | 00:59 |
| anteaya | we have the monday off so can eat anytime from friday to monday | 00:59 |
| anteaya | so I had to hit 3 this time | 01:00 |
| Clint | clarkb: do you not end up with leftovers that last past the weekend? | 01:00 |
| clarkb | Clint: usually enough leftovers for a couple turkey sandwiches | 01:01 |
| *** btully has joined #openstack-infra | 01:01 | |
| clarkb | Clint: I have a huge extended family and things get split up pretty well | 01:01 |
| Clint | this is a kind of efficiency with which i am unfamiliar | 01:01 |
| *** thorst has quit IRC | 01:01 | |
| clarkb | my grandparents had 10 kids and they all had kids and then we all go to my grandmothers | 01:01 |
| anteaya | smoke a turkey then | 01:02 |
| clarkb | and I suppose more of the grandchildren have kids now too o_O (myself included) | 01:02 |
| anteaya | that is a lot of people | 01:02 |
| anteaya | the sick neighbour I spent time with all weekend had a grateful daughter who came up last night | 01:02 |
| anteaya | no idea she could cook but oh my goodness can she | 01:03 |
| anteaya | every once in a while I make a good decision | 01:03 |
| anteaya | baked califlour | 01:03 |
| anteaya | I had never had it before but will be cooking it myself soon | 01:03 |
| clarkb | however one thing there is always plenty of is oyster stuffing because no one likes it but regular stuffing is gone right away so you learn to eat the oyster stuff fast if you want stuffing | 01:04 |
| Clint | haha | 01:04 |
| anteaya | ha ha ha | 01:04 |
| *** heyongli has quit IRC | 01:04 | |
| *** MarkAtwood has quit IRC | 01:05 | |
| anteaya | my father's family had six children, so for breakfast my grandmother would make the food including toast | 01:05 |
| clarkb | mordred: btw I agree with you. 100k nodes | 01:05 |
| *** btully has quit IRC | 01:05 | |
| *** Sukhdev has quit IRC | 01:05 | |
| *** heyongli has joined #openstack-infra | 01:05 | |
| anteaya | my aunt learned to start with a half piece in order to get a follow up full piece | 01:05 |
| clarkb | mordred: then we can have 50 node gate runs and no one will care :) | 01:05 |
| anteaya | if she started with a full piece my grandmother would be finished cooking and no more toast | 01:06 |
| clarkb | anteaya: yes large family eating is a learned skill. Larissa still hasn't learned all the tricks | 01:06 |
| anteaya | are you giving her any tips? | 01:07 |
| timrc | oyster stuffing... that sounds like a cruel joke. | 01:08 |
| *** DericHorn-HP has quit IRC | 01:08 | |
| anteaya | timrc: are you following along, if you get any remember to eat it fast | 01:09 |
| clarkb | timrc: its actually pretty good. Similar to stuffing with the turkey offal in it | 01:09 |
| clarkb | saltier but has a similar richness | 01:09 |
| *** woodster_ has quit IRC | 01:09 | |
| Clint | my dad used to put oyster sauce in the stuffing | 01:09 |
| dims_ | mriedem: lifeless: jgriffith: https://review.openstack.org/#/c/233870/ ping | 01:09 |
| *** geoffarn_ has joined #openstack-infra | 01:11 | |
| anteaya | I have to go to sleep now so I can wake up in the middle of the night and chair a meeting | 01:12 |
| anteaya | night y'all | 01:12 |
| *** geoffarnold has quit IRC | 01:12 | |
| clarkb | good night | 01:13 |
| *** heyongli has quit IRC | 01:15 | |
| *** heyongli has joined #openstack-infra | 01:16 | |
| *** yamamoto has quit IRC | 01:17 | |
| *** hdd has joined #openstack-infra | 01:17 | |
| *** ajmiller has quit IRC | 01:25 | |
| *** heyongli has quit IRC | 01:25 | |
| *** heyongli has joined #openstack-infra | 01:26 | |
| *** gildub has joined #openstack-infra | 01:27 | |
| *** ajmiller has joined #openstack-infra | 01:31 | |
| *** achanda has joined #openstack-infra | 01:31 | |
| *** bgmccollum has joined #openstack-infra | 01:31 | |
| *** ajmiller_ has joined #openstack-infra | 01:32 | |
| *** baoli has joined #openstack-infra | 01:32 | |
| *** geoffarn_ has quit IRC | 01:33 | |
| *** geoffarnold has joined #openstack-infra | 01:33 | |
| *** ajmiller_ has quit IRC | 01:33 | |
| *** zhenguo has joined #openstack-infra | 01:33 | |
| *** mtanino has quit IRC | 01:34 | |
| *** heyongli has quit IRC | 01:35 | |
| *** ajmiller has quit IRC | 01:35 | |
| *** heyongli has joined #openstack-infra | 01:36 | |
| *** DericHorn-HP has joined #openstack-infra | 01:36 | |
| *** sridhar_ram has quit IRC | 01:39 | |
| lifeless | dims_: jfd | 01:42 |
| *** tlian has quit IRC | 01:45 | |
| *** heyongli has quit IRC | 01:45 | |
| *** glauco has quit IRC | 01:46 | |
| *** ivar-laz_ has joined #openstack-infra | 01:46 | |
| *** heyongli has joined #openstack-infra | 01:46 | |
| *** ivar-lazzaro has quit IRC | 01:49 | |
| *** ivar-laz_ has quit IRC | 01:50 | |
| *** stevemar_ has quit IRC | 01:53 | |
| *** heyongli has quit IRC | 01:56 | |
| *** heyongli has joined #openstack-infra | 01:57 | |
| *** unicell1 has quit IRC | 02:02 | |
| *** Piet has joined #openstack-infra | 02:03 | |
| *** yamamoto has joined #openstack-infra | 02:04 | |
| *** DericHorn-HP has quit IRC | 02:05 | |
| *** baoli has quit IRC | 02:05 | |
| *** heyongli has quit IRC | 02:06 | |
| *** heyongli has joined #openstack-infra | 02:07 | |
| *** DericHorn-HP has joined #openstack-infra | 02:10 | |
| openstackgerrit | Merged openstack/requirements: Require oslo.cache 0.8.0 https://review.openstack.org/233689 | 02:12 |
| *** geoffarnold has quit IRC | 02:15 | |
| *** geoffarnold has joined #openstack-infra | 02:15 | |
| *** heyongli has quit IRC | 02:16 | |
| *** heyongli has joined #openstack-infra | 02:17 | |
| *** inteq has joined #openstack-infra | 02:18 | |
| openstackgerrit | Boris Pavlovic proposed openstack-infra/project-config: Fix rally-cue-dsvm job https://review.openstack.org/233909 | 02:21 |
| *** shashank_hegde has quit IRC | 02:24 | |
| *** glauco_ has joined #openstack-infra | 02:24 | |
| *** ianw has quit IRC | 02:25 | |
| *** heyongli has quit IRC | 02:26 | |
| *** heyongli has joined #openstack-infra | 02:27 | |
| *** links has joined #openstack-infra | 02:28 | |
| *** dims_ has quit IRC | 02:28 | |
| *** glauco_ has quit IRC | 02:29 | |
| *** derekh has quit IRC | 02:29 | |
| *** yamahata has quit IRC | 02:29 | |
| *** DericHorn-HP has quit IRC | 02:34 | |
| openstackgerrit | Ken'ichi Ohmichi proposed openstack/requirements: Bump minimum tempest-lib version https://review.openstack.org/233915 | 02:35 |
| *** geoffarnold has quit IRC | 02:36 | |
| *** geoffarnold has joined #openstack-infra | 02:36 | |
| *** heyongli has quit IRC | 02:37 | |
| *** heyongli has joined #openstack-infra | 02:38 | |
| *** salv-orlando has joined #openstack-infra | 02:39 | |
| *** DericHorn-HP has joined #openstack-infra | 02:39 | |
| *** mriedem has quit IRC | 02:39 | |
| *** dims__ has joined #openstack-infra | 02:41 | |
| tonyb | I'm getting "This change or one of its cross-repo dependencies was unable to be automatically merged ...." but I can't see any problems when I rebase on top of master. Is it possible to get logs so I can see what's going on? | 02:41 |
| *** stevemar_ has joined #openstack-infra | 02:42 | |
| openstackgerrit | fumihiko kakuma proposed openstack-infra/project-config: Add job for the ovs mechanism driver with native mode https://review.openstack.org/221143 | 02:43 |
| *** oomichi has joined #openstack-infra | 02:44 | |
| *** carl_baldwin has joined #openstack-infra | 02:46 | |
| *** heyongli has quit IRC | 02:47 | |
| *** MarkAtwood has joined #openstack-infra | 02:47 | |
| *** salv-orlando has quit IRC | 02:47 | |
| oomichi | hello, infra-cores | 02:48 |
| *** heyongli has joined #openstack-infra | 02:48 | |
| tonyb | Where do I report a bug for infra-manual? or should I just push a review with my suggested fix? | 02:48 |
| tonyb | (that's unrelated to my previous question) | 02:48 |
| *** btully has joined #openstack-infra | 02:49 | |
| StevenK | 400: GET /api/v1/projects/search: Unknown argument: "offset" | 02:49 |
| StevenK | Handy, thanks Storyboard | 02:49 |
| *** puranamr has joined #openstack-infra | 02:53 | |
| *** btully has quit IRC | 02:53 | |
| *** hdd has quit IRC | 02:56 | |
| oomichi | can someone take a look at https://review.openstack.org/#/c/233915/ ? | 02:56 |
| oomichi | it gets tempest PTL's +1 already | 02:56 |
| *** heyongli has quit IRC | 02:57 | |
| *** geoffarnold has quit IRC | 02:57 | |
| *** geoffarnold has joined #openstack-infra | 02:58 | |
| *** heyongli has joined #openstack-infra | 02:58 | |
| *** claudiub has quit IRC | 02:58 | |
| *** rguillebert has quit IRC | 02:58 | |
| *** dims__ has quit IRC | 02:59 | |
| *** spzala has quit IRC | 03:03 | |
| StevenK | tonyb: I can't see infra-manual in either storyboard or launchpad. I don't think it fits under openstack-manuals either, so push up a branch? | 03:06 |
| asselin__ | tonyb, yea, best to just fix it | 03:06 |
| tonyb | mmkay. | 03:07 |
| asselin__ | tonyb, StevenK I think it's in system-config...let me check | 03:07 |
| StevenK | Maybe we're being trolled, and infra-manual is in phabricator already. | 03:07 |
| * StevenK glances at ruagair | 03:07 | |
| *** heyongli has quit IRC | 03:07 | |
| *** jhesketh has quit IRC | 03:08 | |
| *** salv-orlando has joined #openstack-infra | 03:08 | |
| asselin__ | tonyb, http://git.openstack.org/cgit/openstack-infra/system-config/tree/doc/source contain what you're trying to fix? | 03:08 |
| *** heyongli has joined #openstack-infra | 03:08 | |
| *** puranamr has quit IRC | 03:08 | |
| tonyb | asselin__: http://git.openstack.org/cgit/openstack-infra/infra-manual/tree/doc/source/developers.rst | 03:09 |
| tonyb | asselin__: it's not a bug as such but I know >5 people that haven't been able to get a working setup without people "in the know" helping them. | 03:10 |
| tonyb | asselin__: so I figure they could be a little clearer | 03:10 |
| asselin__ | tonyb, ok cool, yeah best to just improve it | 03:11 |
| tonyb | asselin__: not laying blame just trying to do my bit :) | 03:11 |
| asselin__ | tonyb, regarding this "I'm getting "This change or one of its cross-repo dependencies was unable to be automatically merged ...." but I can't see any problems when I rebase on top of master. Is it possible to get logs so I can see what's going on?" | 03:12 |
| asselin__ | jeblair, did restart one of the zuul mergers due to no inodes | 03:12 |
| asselin__ | so if it's not recent, just do a recheck. | 03:12 |
| *** jhesketh has joined #openstack-infra | 03:13 | |
| tonyb | asselin__: the jenkins message was about 6 hours ago | 03:13 |
| tonyb | asselin__: was that before the restart? | 03:14 |
| *** otter768 has quit IRC | 03:14 | |
| asselin__ | that sounds about the same time | 03:14 |
| tonyb | asselin__: okay I'll recheck | 03:14 |
| asselin__ | ok, heading off, goodnight | 03:15 |
| *** otter768 has joined #openstack-infra | 03:15 | |
| *** shashank_hegde has joined #openstack-infra | 03:15 | |
| *** salv-orlando has quit IRC | 03:17 | |
| *** puranamr has joined #openstack-infra | 03:17 | |
| *** heyongli has quit IRC | 03:18 | |
| *** glauco_ has joined #openstack-infra | 03:18 | |
| *** heyongli has joined #openstack-infra | 03:18 | |
| oomichi | tonyb: thanks for your review :) | 03:20 |
| tonyb | oomichi: np sorry I can't +2 it ;P | 03:21 |
| oomichi | tonyb: no problem, your score will invite the other +2s :-) | 03:22 |
| tonyb | oomichi: :) | 03:22 |
| *** mdbooth has quit IRC | 03:22 | |
| *** glauco_ has quit IRC | 03:23 | |
| tonyb | oomichi: I watch the requirements repo and try to look at all the changes but yours didn't generate an email (yet)? strange. | 03:23 |
| *** puranamr has quit IRC | 03:23 | |
| *** guardianL has joined #openstack-infra | 03:24 | |
| *** guardianL has left #openstack-infra | 03:25 | |
| *** heyongli has quit IRC | 03:28 | |
| *** mdbooth has joined #openstack-infra | 03:28 | |
| *** heyongli has joined #openstack-infra | 03:29 | |
| openstackgerrit | Boris Pavlovic proposed openstack-infra/project-config: Fix rally-cue-dsvm job https://review.openstack.org/233909 | 03:29 |
| *** shashank_hegde has quit IRC | 03:32 | |
| *** hdd has joined #openstack-infra | 03:33 | |
| *** mrmartin has joined #openstack-infra | 03:33 | |
| *** heyongli has quit IRC | 03:38 | |
| *** heyongli has joined #openstack-infra | 03:39 | |
| *** fitoduarte has quit IRC | 03:41 | |
| *** amotoki has joined #openstack-infra | 03:44 | |
| *** shashank_hegde has joined #openstack-infra | 03:47 | |
| oomichi | tonyb: oh, strange. I'm not sure the reason, maybe I have configured "CC Me On Comments I Write" on my gerrit setting | 03:47 |
| oomichi | tonyb: I could not imagine the other factors.. :( | 03:47 |
| tonyb | oomichi: Oh it could be the mail is stick in a mail router somewhere. | 03:48 |
| *** heyongli has quit IRC | 03:48 | |
| *** heyongli has joined #openstack-infra | 03:49 | |
| *** _denisra has quit IRC | 03:50 | |
| *** amotoki has quit IRC | 03:51 | |
| *** opal has joined #openstack-infra | 03:52 | |
| *** amotoki has joined #openstack-infra | 03:55 | |
| *** aeng has quit IRC | 03:57 | |
| *** aeng has joined #openstack-infra | 03:57 | |
| *** carl_baldwin has quit IRC | 03:58 | |
| *** links has quit IRC | 03:58 | |
| *** heyongli has quit IRC | 03:59 | |
| *** dchen has joined #openstack-infra | 03:59 | |
| *** heyongli has joined #openstack-infra | 03:59 | |
| *** dims__ has joined #openstack-infra | 03:59 | |
| *** harlowja_at_home has joined #openstack-infra | 04:00 | |
| *** opal has left #openstack-infra | 04:02 | |
| *** megm has quit IRC | 04:02 | |
| *** Sree has joined #openstack-infra | 04:03 | |
| *** tiswanso has quit IRC | 04:03 | |
| *** megm has joined #openstack-infra | 04:03 | |
| *** Sree has quit IRC | 04:04 | |
| *** Sree has joined #openstack-infra | 04:04 | |
| *** dims__ has quit IRC | 04:06 | |
| *** heyongli has quit IRC | 04:09 | |
| *** heyongli has joined #openstack-infra | 04:10 | |
| *** salv-orlando has joined #openstack-infra | 04:15 | |
| *** dchen has quit IRC | 04:16 | |
| oomichi | jhesketh: hello | 04:17 |
| *** amotoki has quit IRC | 04:18 | |
| *** heyongli has quit IRC | 04:19 | |
| *** salv-orlando has quit IRC | 04:19 | |
| *** heyongli has joined #openstack-infra | 04:20 | |
| *** Sukhdev has joined #openstack-infra | 04:20 | |
| timrc | mordred: lol, alias switch-cloud='python -c "from __future__ import print_function;import os_client_config as c, sys; [print(\"export OS_%s=%s\" % (a.upper(), b)) for (a,b) in c.OpenStackConfig().get_one_cloud(sys.argv[1]).auth.iteritems()]"' | 04:20 |
| timrc | mordred: Would use like so $(switch-cloud foo) | 04:21 |
| * timrc feels awful now :) | 04:21 | |
| jhesketh | oomichi: howdy | 04:21 |
| openstackgerrit | Sirushti Murugesan proposed openstack-infra/devstack-gate: Copy grenade saved files into the log directory https://review.openstack.org/233369 | 04:21 |
| oomichi | jhesketh: can you take a look at https://review.openstack.org/#/c/233915/ ? | 04:22 |
| oomichi | jhesketh: that is g-r patch for tempest-lib, and mtreinish +1 for that :) | 04:22 |
| *** geoffarn_ has joined #openstack-infra | 04:23 | |
| *** geoffarnold has quit IRC | 04:24 | |
| jhesketh | oomichi: will take a look | 04:26 |
| jhesketh | lgtm | 04:27 |
| oomichi | jhesketh: thanks in advance :) | 04:27 |
| *** amotoki has joined #openstack-infra | 04:29 | |
| *** amotoki_ has joined #openstack-infra | 04:29 | |
| *** heyongli has quit IRC | 04:29 | |
| *** niska has quit IRC | 04:30 | |
| *** heyongli has joined #openstack-infra | 04:30 | |
| harlowja_at_home | ok whose gonna setup a scale lab, lol | 04:32 |
| *** Sree_ has joined #openstack-infra | 04:36 | |
| *** Sree_ is now known as Guest81197 | 04:36 | |
| *** salv-orlando has joined #openstack-infra | 04:36 | |
| *** amotoki_ has quit IRC | 04:36 | |
| *** btully has joined #openstack-infra | 04:38 | |
| *** niska has joined #openstack-infra | 04:38 | |
| *** naohirot has joined #openstack-infra | 04:39 | |
| *** Sree has quit IRC | 04:39 | |
| *** nikhil_k has joined #openstack-infra | 04:39 | |
| *** heyongli has quit IRC | 04:40 | |
| naohirot | infra team | 04:40 |
| *** salv-orlando has quit IRC | 04:40 | |
| *** heyongli has joined #openstack-infra | 04:40 | |
| naohirot | would you please review and approve https://review.openstack.org/#/c/218097/ ? | 04:40 |
| naohirot | As a fact, I'm waining for 1.5 months | 04:41 |
| *** mmedvede has quit IRC | 04:41 | |
| *** Piet has quit IRC | 04:41 | |
| naohirot | please let me go to the next step, thanks in advance. | 04:41 |
| *** mmedvede has joined #openstack-infra | 04:42 | |
| *** nikhil has quit IRC | 04:42 | |
| *** dosaboy has quit IRC | 04:42 | |
| *** skraynev has quit IRC | 04:42 | |
| *** dosaboy has joined #openstack-infra | 04:43 | |
| *** dchen has joined #openstack-infra | 04:43 | |
| *** skraynev has joined #openstack-infra | 04:43 | |
| *** aviau_ has joined #openstack-infra | 04:43 | |
| *** geoffarnold has joined #openstack-infra | 04:44 | |
| *** aviau has quit IRC | 04:44 | |
| *** geoffarn_ has quit IRC | 04:44 | |
| *** harlowja_at_home has quit IRC | 04:45 | |
| *** Piet has joined #openstack-infra | 04:46 | |
| *** asettle has quit IRC | 04:49 | |
| *** heyongli has quit IRC | 04:50 | |
| *** heyongli has joined #openstack-infra | 04:51 | |
| *** mestery has joined #openstack-infra | 04:51 | |
| *** heyongli has quit IRC | 05:00 | |
| *** heyongli has joined #openstack-infra | 05:01 | |
| *** mestery has quit IRC | 05:02 | |
| *** ybathia has joined #openstack-infra | 05:02 | |
| *** links has joined #openstack-infra | 05:06 | |
| *** glauco has joined #openstack-infra | 05:06 | |
| *** heyongli has quit IRC | 05:10 | |
| *** glauco has quit IRC | 05:11 | |
| *** heyongli has joined #openstack-infra | 05:11 | |
| *** yfried has quit IRC | 05:11 | |
| *** Yingxin2 has joined #openstack-infra | 05:15 | |
| *** Yingxin2 has left #openstack-infra | 05:16 | |
| *** DericHorn-HP has quit IRC | 05:17 | |
| *** mrmartin has quit IRC | 05:17 | |
| *** dchen has quit IRC | 05:18 | |
| *** geoffarnold has quit IRC | 05:19 | |
| *** geoffarnold has joined #openstack-infra | 05:19 | |
| openstackgerrit | Tim Burke proposed openstack-infra/devstack-gate: Fix some log help typos https://review.openstack.org/233942 | 05:20 |
| *** heyongli has quit IRC | 05:21 | |
| *** heyongli has joined #openstack-infra | 05:21 | |
| openstackgerrit | Merged openstack/requirements: pyinotify does not work on windows or osx https://review.openstack.org/231118 | 05:23 |
| *** heyongli has quit IRC | 05:31 | |
| *** Sree has joined #openstack-infra | 05:32 | |
| *** heyongli has joined #openstack-infra | 05:32 | |
| *** asettle has joined #openstack-infra | 05:33 | |
| *** Guest81197 has quit IRC | 05:35 | |
| *** Sukhdev_ has joined #openstack-infra | 05:35 | |
| *** Sukhdev_ has quit IRC | 05:36 | |
| *** Sukhdev has quit IRC | 05:37 | |
| *** unicell has joined #openstack-infra | 05:38 | |
| *** HeOS has joined #openstack-infra | 05:38 | |
| *** geoffarnold has quit IRC | 05:40 | |
| *** geoffarnold has joined #openstack-infra | 05:40 | |
| *** nikhil has joined #openstack-infra | 05:41 | |
| *** hdd has quit IRC | 05:41 | |
| *** heyongli has quit IRC | 05:41 | |
| *** heyongli has joined #openstack-infra | 05:42 | |
| *** nikhil_k has quit IRC | 05:43 | |
| *** heyongli has quit IRC | 05:51 | |
| *** heyongli has joined #openstack-infra | 05:52 | |
| *** AJaeger has joined #openstack-infra | 05:56 | |
| *** luqas has joined #openstack-infra | 05:58 | |
| *** sabeen1 has joined #openstack-infra | 05:58 | |
| *** glauco has joined #openstack-infra | 06:00 | |
| *** heyongli has quit IRC | 06:02 | |
| *** matbu has quit IRC | 06:02 | |
| *** heyongli has joined #openstack-infra | 06:02 | |
| *** stevemar_ has quit IRC | 06:03 | |
| *** dims__ has joined #openstack-infra | 06:03 | |
| *** matbu has joined #openstack-infra | 06:03 | |
| *** flepied has quit IRC | 06:05 | |
| *** glauco has quit IRC | 06:05 | |
| *** jtomasek has joined #openstack-infra | 06:07 | |
| *** lewo has joined #openstack-infra | 06:07 | |
| *** sputnik13 has joined #openstack-infra | 06:08 | |
| *** dims__ has quit IRC | 06:09 | |
| *** asettle has quit IRC | 06:11 | |
| *** sputnik13 has quit IRC | 06:11 | |
| *** heyongli has quit IRC | 06:12 | |
| *** heyongli has joined #openstack-infra | 06:13 | |
| openstackgerrit | Mehdi Abaakouk (sileht) proposed openstack/requirements: Raise jsonpath-rw-ext version https://review.openstack.org/224494 | 06:14 |
| *** links has quit IRC | 06:20 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/231327 | 06:21 |
| openstackgerrit | Merged openstack-infra/lodgeit: Fix header in help screen https://review.openstack.org/232405 | 06:21 |
| *** openstack has joined #openstack-infra | 06:30 | |
| *** amotoki has joined #openstack-infra | 06:30 | |
| openstackgerrit | fumihiko kakuma proposed openstack-infra/project-config: Add job for the ovs mechanism driver with native mode https://review.openstack.org/221143 | 06:31 |
| *** shashank_hegde has quit IRC | 06:31 | |
| *** heyongli has quit IRC | 06:32 | |
| *** heyongli has joined #openstack-infra | 06:33 | |
| *** shashank_hegde has joined #openstack-infra | 06:34 | |
| *** scheuran has joined #openstack-infra | 06:35 | |
| openstackgerrit | Jean-Emile DARTOIS proposed openstack-infra/project-config: Add watcher-specs https://review.openstack.org/233679 | 06:37 |
| *** dkranz has joined #openstack-infra | 06:38 | |
| *** flepied has joined #openstack-infra | 06:39 | |
| *** ParsectiX has joined #openstack-infra | 06:42 | |
| *** heyongli has quit IRC | 06:43 | |
| *** _denisra has joined #openstack-infra | 06:43 | |
| *** heyongli has joined #openstack-infra | 06:43 | |
| *** geoffarnold has quit IRC | 06:44 | |
| *** peristeri has joined #openstack-infra | 06:44 | |
| *** asmigala has joined #openstack-infra | 06:44 | |
| *** geoffarnold has joined #openstack-infra | 06:44 | |
| *** gildub has quit IRC | 06:46 | |
| oomichi | jeblair: hello | 06:46 |
| *** achanda has quit IRC | 06:46 | |
| *** ybathia has quit IRC | 06:47 | |
| *** achanda has joined #openstack-infra | 06:48 | |
| *** abregman has joined #openstack-infra | 06:50 | |
| *** nmagnezi has joined #openstack-infra | 06:51 | |
| *** yfried has joined #openstack-infra | 06:51 | |
| *** mrmartin has joined #openstack-infra | 06:52 | |
| *** sshnaidm has joined #openstack-infra | 06:52 | |
| *** heyongli has quit IRC | 06:53 | |
| *** k4n0 has joined #openstack-infra | 06:53 | |
| *** amotoki has quit IRC | 06:54 | |
| *** heyongli has joined #openstack-infra | 06:54 | |
| *** glauco has joined #openstack-infra | 06:55 | |
| *** yfried has quit IRC | 06:55 | |
| *** yfried has joined #openstack-infra | 06:56 | |
| yolanda | morning | 06:56 |
| *** e0ne has joined #openstack-infra | 06:58 | |
| *** e0ne has quit IRC | 06:59 | |
| *** glauco has quit IRC | 06:59 | |
| openstackgerrit | Egor Guz proposed openstack/requirements: Update Python client for Docker https://review.openstack.org/233992 | 07:02 |
| *** heyongli has quit IRC | 07:03 | |
| *** heyongli has joined #openstack-infra | 07:04 | |
| *** sabeen1 has quit IRC | 07:04 | |
| *** dims__ has joined #openstack-infra | 07:05 | |
| openstackgerrit | Merged openstack-infra/system-config: Use SNI/SAN on static.openstack.org https://review.openstack.org/232748 | 07:07 |
| *** dims__ has quit IRC | 07:10 | |
| oomichi | yolanda: morning | 07:10 |
| oomichi | yolanda: can you take a look at https://review.openstack.org/#/c/233915/ ? that will help our tempest work | 07:11 |
| oomichi | yolanda: sorry for catching you in morning | 07:11 |
| yolanda | hi oomichi | 07:11 |
| *** links has joined #openstack-infra | 07:13 | |
| *** heyongli has quit IRC | 07:13 | |
| *** heyongli has joined #openstack-infra | 07:14 | |
| *** armax has quit IRC | 07:15 | |
| yolanda | AJaeger, you there ? as it's a change to deal with requirements, i'd like to have your feedback as well, for oomichi's request | 07:15 |
| AJaeger | hi yolanda | 07:16 |
| AJaeger | let me check... | 07:16 |
| *** e0ne has joined #openstack-infra | 07:16 | |
| *** btully has quit IRC | 07:16 | |
| AJaeger | yolanda: for those changes it's important that you have experts weight in - mtreinish's +1 here was crucial ;) | 07:17 |
| AJaeger | yolanda: the requirements repo in general has it's own core team that is active, infra policy in the past has been to only approve in case of emergencies but not do normal reviews. So, pinging lifeless or dhellmann would be the preferred solution. | 07:19 |
| yolanda | AJaeger, good to know | 07:19 |
| *** dizquierdo has joined #openstack-infra | 07:19 | |
| *** achanda has quit IRC | 07:20 | |
| AJaeger | yolanda, oomichi : I don't consider this change as urgent, it was just proposed a few hours ago. I understand it blocks work. | 07:20 |
| AJaeger | yolanda: With mtreinish's +1, no worries IMHO - just my comments for the future;) But talk with fungi and get his feel for when to engage or not. I'm not infra-root and cannot give autoritative guidance ;) | 07:21 |
| *** jhesketh has quit IRC | 07:22 | |
| oomichi | yolanda: AJaeger: thanks for your helps, very glad to merge it :) | 07:23 |
| *** jhesketh has joined #openstack-infra | 07:23 | |
| *** heyongli has quit IRC | 07:24 | |
| *** zeih has joined #openstack-infra | 07:24 | |
| *** heyongli has joined #openstack-infra | 07:24 | |
| yolanda | AJaeger, thanks for the feedback there | 07:26 |
| openstackgerrit | Khai Do proposed openstack-infra/jenkins-job-builder: fix for python 3, decode job output https://review.openstack.org/233265 | 07:26 |
| *** geoffarn_ has joined #openstack-infra | 07:27 | |
| *** bexelbie_ has joined #openstack-infra | 07:27 | |
| naohirot | AJaeger: hello | 07:27 |
| openstackgerrit | Merged openstack-infra/system-config: Remove jenkins & nodepool creds from nodepool.yaml https://review.openstack.org/199750 | 07:29 |
| naohirot | AJaeger: I'm working in Ironic. I had a conversion with you before in Ironic gerrit review. could you review this https://review.openstack.org/#/c/218097/ ? | 07:29 |
| *** matrohon has joined #openstack-infra | 07:29 | |
| AJaeger | naohirot: will do... | 07:29 |
| naohirot | AJaeger: I'm waiting its approval for 1.5 months. Yeah, please thanks. | 07:30 |
| *** geoffarnold has quit IRC | 07:31 | |
| AJaeger | naohirot: sorry, that patch fell through ;( | 07:31 |
| naohirot | AJaeger: fell through? it's not good? | 07:32 |
| AJaeger | naohirot: we missed to review it | 07:32 |
| AJaeger | naohirot: will this repo be part of Ironic? | 07:32 |
| *** rguillebert has joined #openstack-infra | 07:32 | |
| *** yolanda has quit IRC | 07:32 | |
| naohirot | AJaeger: I see, my patch fell through core team's eyes. I got it. :0) | 07:33 |
| AJaeger | naohirot: yeah ;( | 07:33 |
| AJaeger | naohirot: I left a question for you on the review - please answer it there so that others see the answer as well... | 07:33 |
| *** yolanda has joined #openstack-infra | 07:34 | |
| *** amotoki has joined #openstack-infra | 07:34 | |
| *** heyongli has quit IRC | 07:34 | |
| *** luqas__ has quit IRC | 07:34 | |
| naohirot | AJaeger: this repo is not Ironic official, but Ironic Vendor Driver repo. | 07:34 |
| naohirot | AJaeger: Okay. | 07:34 |
| *** heyongli has joined #openstack-infra | 07:35 | |
| AJaeger | naohirot: Ah, ok. Just asked because Jim's comment was not clear. I'll add another comment | 07:35 |
| *** shashank_hegde has quit IRC | 07:36 | |
| *** jed56 has quit IRC | 07:36 | |
| naohirot | AJaeger: Okay. all of Ironic driver repo is not Ironic official which is the way of Ironic I believe. | 07:36 |
| *** sridhar_ram has joined #openstack-infra | 07:37 | |
| AJaeger | naohirot: every team handles this different ;) | 07:37 |
| *** mpavone has joined #openstack-infra | 07:37 | |
| AJaeger | naohirot: what about the releases: Do you want to import those as well? You cannot delete them afterwards... | 07:37 |
| openstackgerrit | Sergey Nikitin proposed openstack/requirements: Updated oslo.db to 3.0.0 https://review.openstack.org/234015 | 07:37 |
| naohirot | AJaeger: Yeah, I hope driver will be official like Neutron. | 07:37 |
| naohirot | AJaeger: what does it mean by "import"? | 07:39 |
| naohirot | AJaeger: what will not be able to delete? | 07:39 |
| naohirot | AJaeger: s/delete/be deleted by me/ | 07:40 |
| *** yamahata has joined #openstack-infra | 07:40 | |
| naohirot | AJaeger: do you mean release tag which I made already 0.1 and 0.2? | 07:41 |
| AJaeger | naohirot: we import your upstream repository. If you want to delete in the future any tags (there are none) or release, it's too late. You have to cleanup now - or say the repo is fine including the release. | 07:41 |
| AJaeger | naohirot: Yes. If you don't want them, remove them. If you want them, leave them in. Your call. | 07:41 |
| *** sridhar_ram has quit IRC | 07:41 | |
| AJaeger | Just wanted to point out that you can't change your mind ;) | 07:41 |
| *** arxcruz has joined #openstack-infra | 07:41 | |
| naohirot | AJaeger: I'd like you to import all of them including the tag 0.1 and 0.2. | 07:42 |
| *** cdent has joined #openstack-infra | 07:42 | |
| AJaeger | naohirot: state it as review comment and I'm happy to +2 ;) | 07:42 |
| *** amotoki has quit IRC | 07:42 | |
| cdent | gerrit is very sleep this morning | 07:42 |
| cdent | sleepy! (me too apparently) | 07:42 |
| *** sergmelikyan has joined #openstack-infra | 07:43 | |
| loquacities | yeah, slowed down for me about an hour ago | 07:43 |
| AJaeger | yolanda: do you want to review naohirot's change as well? We missed to review it - 218097 and I'd like to make amends if possible ;) | 07:43 |
| naohirot | AJaeger: thanks. I'v release 0.1 for kilo, 0.2 for liberty. So not being able to remove is fine for me. | 07:43 |
| yolanda | AJaeger, sure | 07:43 |
| *** e0ne has quit IRC | 07:44 | |
| *** heyongli has quit IRC | 07:44 | |
| naohirot | yolanda: AJaeger: thanks! | 07:45 |
| AJaeger | you're welcome, naohirot | 07:45 |
| *** heyongli has joined #openstack-infra | 07:45 | |
| spredzy | yolanda, is there any current/planed effort to move from openstack-infra/puppet-httpd to puppetlabs/apache for openstack-infra/puppet-* modules ? | 07:45 |
| *** ianw has joined #openstack-infra | 07:45 | |
| yolanda | spredzy, not in short term | 07:46 |
| yolanda | there wer some discussions about it, but for the moment we'll continue with puppet-httpd | 07:46 |
| spredzy | yolanda, thanks. Also same question for puppetlabs-mysql ? Any current/planned effort to bump it up to a newer version ? | 07:47 |
| yolanda | spredzy, no movements there as well. What are the benefits to upgrade puppetlabs-mysql version from our current one? we could open a discussion if needed | 07:48 |
| *** geoffarn_ has quit IRC | 07:48 | |
| *** geoffarnold has joined #openstack-infra | 07:48 | |
| *** glauco has joined #openstack-infra | 07:49 | |
| spredzy | yolanda, current version is 0.6.1 (https://github.com/openstack-infra/system-config/blob/master/modules.env#L57) which does not have support out of the box for EL7/C7 platforms (MariaDB default) https://github.com/puppetlabs/puppetlabs-mysql/blob/0.6.1/manifests/params.pp | 07:50 |
| *** amotoki has joined #openstack-infra | 07:50 | |
| *** persia has quit IRC | 07:50 | |
| yolanda | spredzy, i think you can send that topic to openstack-infra mailing list, so we can have open discussion there, about pros/cons | 07:51 |
| spredzy | while a more recent version of the module would bring that support https://github.com/puppetlabs/puppetlabs-mysql/blob/3.6.1/manifests/params.pp#L45-L50 | 07:51 |
| openstackgerrit | Merged openstack-infra/project-config: Add job for publication Murano docs on docs.openstack.org https://review.openstack.org/230359 | 07:51 |
| *** otter768 has quit IRC | 07:51 | |
| *** persia has joined #openstack-infra | 07:51 | |
| spredzy | yolanda, sure will do. | 07:51 |
| *** glauco has quit IRC | 07:53 | |
| *** heyongli has quit IRC | 07:54 | |
| *** ifarkas has joined #openstack-infra | 07:54 | |
| *** heyongli has joined #openstack-infra | 07:55 | |
| *** salv-orlando has joined #openstack-infra | 07:59 | |
| *** bexelbie_ is now known as bexelbie | 07:59 | |
| *** markus_z has joined #openstack-infra | 08:01 | |
| *** jordanP has joined #openstack-infra | 08:02 | |
| *** salv-orlando has quit IRC | 08:03 | |
| *** jistr has joined #openstack-infra | 08:03 | |
| *** yfried is now known as yfried|afk | 08:04 | |
| *** btully has joined #openstack-infra | 08:04 | |
| *** heyongli has quit IRC | 08:05 | |
| *** heyongli has joined #openstack-infra | 08:05 | |
| *** dims__ has joined #openstack-infra | 08:06 | |
| *** jcoufal has joined #openstack-infra | 08:06 | |
| *** claudiub has joined #openstack-infra | 08:06 | |
| *** yfried|afk is now known as yfried | 08:06 | |
| *** links has quit IRC | 08:08 | |
| *** geoffarnold has quit IRC | 08:09 | |
| *** btully has quit IRC | 08:09 | |
| *** geoffarnold has joined #openstack-infra | 08:10 | |
| *** hashar has joined #openstack-infra | 08:10 | |
| *** amotoki has quit IRC | 08:10 | |
| *** dims__ has quit IRC | 08:11 | |
| *** derekh has joined #openstack-infra | 08:11 | |
| anteaya | so gerrit, cpu usage is way up: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=25&rra_id=all | 08:13 |
| anteaya | as is used space on / http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=32&rra_id=all | 08:13 |
| anteaya | I wonder what javamelody says is happening with garbage collection | 08:13 |
| *** sergmelikyan has quit IRC | 08:14 | |
| anteaya | plus I wonder if our log rotation is actually rotating logs | 08:14 |
| yolanda | hi anteaya, i was just checking with jhesketh, about the gerrit slowness | 08:14 |
| anteaya | oh? | 08:15 |
| jhesketh | o/ | 08:15 |
| *** heyongli has quit IRC | 08:15 | |
| anteaya | I didn't see that in channel | 08:15 |
| anteaya | hello jhesketh | 08:15 |
| anteaya | how are you this fine day? | 08:15 |
| jhesketh | good thank you | 08:15 |
| yolanda | yep, i saw the concerns in channel, and also gerrit is slow for me, so i started looking | 08:15 |
| *** heyongli has joined #openstack-infra | 08:16 | |
| *** amotoki has joined #openstack-infra | 08:17 | |
| anteaya | well that is good to know | 08:17 |
| *** sergmelikyan has joined #openstack-infra | 08:17 | |
| eantyshev | jeblair: Hello! May I ask you opinion on https://review.openstack.org/207094 ? | 08:17 |
| *** eantyshev has left #openstack-infra | 08:17 | |
| *** links has joined #openstack-infra | 08:18 | |
| yolanda | anteaya, so log rotation is working, 30 days are stored for manage projects, for gerrit is even lower now | 08:18 |
| *** svasheka has joined #openstack-infra | 08:18 | |
| anteaya | well something has taken almost all the space on / | 08:19 |
| anteaya | we have 3G to spare | 08:19 |
| yolanda | we may need less than 30 entries, if we fill disk space with those | 08:19 |
| yolanda | 7.3G./apache2 | 08:19 |
| openstackgerrit | Elena Ezhova proposed openstack/requirements: Bump oslo.service version https://review.openstack.org/234026 | 08:20 |
| anteaya | yeah not sure if we need to be carrying 7G of apache logs | 08:20 |
| yolanda | i'm just executing a du to get general space consumption, but apache logs look like a big consumer | 08:21 |
| *** sshnaidm has quit IRC | 08:21 | |
| anteaya | yeah, those numbers are up around pre log rotation numbers | 08:21 |
| anteaya | anyhow I'm back to bed | 08:22 |
| anteaya | good night | 08:22 |
| *** devvesa has joined #openstack-infra | 08:22 | |
| *** yfried is now known as yfried|afk | 08:23 | |
| *** links has quit IRC | 08:24 | |
| *** amotoki has quit IRC | 08:25 | |
| hrubi | Hi, I get this, when trying to submit review to openstack-infra/gearman plugin: "fatal: ICLA contributor agreement requires current contact information.". In the contact information I have Full Name and Preferred Email set, the "offline contact" fields are blank. When I try to fill them, I get "Cannot store contact information". Is this some known problem or am I doing it wrong? | 08:25 |
| *** adrianmo has joined #openstack-infra | 08:25 | |
| *** heyongli has quit IRC | 08:25 | |
| hrubi | for the record, I was able to submit review to openstack-infra/zuul and openstack-infra/jenkins-job-builder, with the same contact info | 08:25 |
| *** garyk has joined #openstack-infra | 08:25 | |
| *** salv-orlando has joined #openstack-infra | 08:26 | |
| *** heyongli has joined #openstack-infra | 08:26 | |
| jhesketh | hrubi: have you registered on the foundation website and signed the CLA there? | 08:26 |
| *** derekh has quit IRC | 08:26 | |
| *** derekh has joined #openstack-infra | 08:29 | |
| *** amotoki has joined #openstack-infra | 08:29 | |
| hrubi | jhesketh: On https://review.openstack.org/#/settings/agreements I can see ICLA with status "Verified" | 08:30 |
| hrubi | so yes, I have signed it | 08:30 |
| jhesketh | hrubi: what do you get here https://www.openstack.org/profile/agreements | 08:31 |
| *** yamahata has quit IRC | 08:31 | |
| *** pelix has joined #openstack-infra | 08:31 | |
| *** oomichi has quit IRC | 08:31 | |
| *** links has joined #openstack-infra | 08:31 | |
| adrianmo | Hi guys, I get an error when I try to update my contact information. I signed the ICLA. And cannot push to upstream without contact info. http://i.imgur.com/N68Mf8r.png | 08:32 |
| *** yfried|afk is now known as yfried | 08:33 | |
| AJaeger | hrubi, adrianmo: Let me cite fungi: | 08:33 |
| hrubi | adrianmo: just trying to solve the same problem.. | 08:33 |
| AJaeger | " it needs your gerrit preferred e-mail address to match a primary e-mail address for a foundation individual member account" | 08:33 |
| AJaeger | "you may want to revisit the second paragraph of http://docs.openstack.org/infra/manual/developers.html#account-setup and make sure you've done that" | 08:34 |
| openstackgerrit | Yanis Guenane proposed openstack-infra/puppet-mysql_backup: Support mysql_backup installation on EL7 derivative https://review.openstack.org/234029 | 08:34 |
| AJaeger | "there's some additional troubleshooting suggestions documented at https://ask.openstack.org/question/56720 if you need" | 08:34 |
| *** openstack has joined #openstack-infra | 08:48 | |
| openstackgerrit | Jan Hruban proposed openstack-infra/gearman-plugin: Fix permission problem when finding builds https://review.openstack.org/234036 | 08:49 |
| *** amotoki has joined #openstack-infra | 08:49 | |
| *** dingyichen has quit IRC | 08:49 | |
| yolanda | bauzas, i see, etherpad very slow as well for me | 08:49 |
| bauzas | yolanda: thanks for confirming | 08:50 |
| bauzas | I wondered if it was a peering problme | 08:50 |
| *** flaper87 has quit IRC | 08:51 | |
| *** flaper87 has joined #openstack-infra | 08:51 | |
| *** mdbooth has joined #openstack-infra | 08:51 | |
| *** geoffarn_ has joined #openstack-infra | 08:52 | |
| *** BharatK_ has quit IRC | 08:52 | |
| *** notnownikki has joined #openstack-infra | 08:53 | |
| *** ZZelle has quit IRC | 08:53 | |
| *** ZZelle has joined #openstack-infra | 08:53 | |
| *** sshnaidm has quit IRC | 08:54 | |
| *** sshnaidm has joined #openstack-infra | 08:54 | |
| *** links has quit IRC | 08:54 | |
| *** links has joined #openstack-infra | 08:54 | |
| *** markus_z has quit IRC | 08:54 | |
| *** markus_z has joined #openstack-infra | 08:54 | |
| *** abregman has quit IRC | 08:54 | |
| *** abregman has joined #openstack-infra | 08:54 | |
| *** heyongli has quit IRC | 08:54 | |
| *** heyongli has joined #openstack-infra | 08:54 | |
| loquacities | bauzas: etherpad is dead http://downforeveryoneorjustme.com/etherpad.openstack.org/ | 08:55 |
| bauzas | awesomeness | 08:55 |
| loquacities | yeah, kinda need it, though :S | 08:55 |
| jhesketh | loquacities: yep, we're aware, taking a look | 08:56 |
| loquacities | jhesketh: cool beans | 08:56 |
| *** heyongli has quit IRC | 08:56 | |
| loquacities | didn't expect you to be still working :P | 08:56 |
| yolanda | i restarted apache on etherpad, it looks better now | 08:56 |
| loquacities | oh, yep, i've got some joy now :) | 08:57 |
| *** heyongli has joined #openstack-infra | 08:57 | |
| *** geoffarnold has quit IRC | 08:57 | |
| jhesketh | yeah just watching top, looks happier | 08:57 |
| openstackgerrit | Yanis Guenane proposed openstack-infra/puppet-lodgeit: Support LodgeIt installation on EL7 derivatives https://review.openstack.org/234032 | 08:57 |
| *** sergmelikyan has joined #openstack-infra | 08:57 | |
| *** marzif has joined #openstack-infra | 08:57 | |
| *** swann has joined #openstack-infra | 08:59 | |
| *** skolekonov has joined #openstack-infra | 08:59 | |
| bauzas | yolanda: thanks | 08:59 |
| yolanda | np | 09:00 |
| samueldmq | morning | 09:00 |
| *** Sree_ has joined #openstack-infra | 09:00 | |
| *** Sree_ is now known as Guest51863 | 09:00 | |
| *** adrianmo has quit IRC | 09:00 | |
| samueldmq | I see a keystone change failing the gate because of a ContextualVersionConflict when parsing requirements | 09:00 |
| openstackgerrit | Yanis Guenane proposed openstack-infra/puppet-lodgeit: Support LodgeIt installation on EL7 derivatives https://review.openstack.org/234032 | 09:01 |
| samueldmq | would anyone be able to give me a hand to debug that ? I wonder whether it is an error in our requirements or not | 09:01 |
| samueldmq | log: http://logs.openstack.org/43/233943/1/check/gate-keystone-python27/ad5279b/console.html#_2015-10-13_05_30_22_252 | 09:02 |
| *** Sree has quit IRC | 09:03 | |
| *** cdent has quit IRC | 09:04 | |
| yolanda | jhesketh, for disk space on gerrit, /dev/xvda1 is at 97%, other volumes, for example the one that is mapping /opt, has free space, can we take some space from there? do you have some process or rules for that? | 09:05 |
| jhesketh | yolanda: you were saying its taken up mostly by logs? | 09:06 |
| *** BharatK_ has joined #openstack-infra | 09:06 | |
| yolanda | yes | 09:06 |
| yolanda | logs and backups | 09:06 |
| *** heyongli has quit IRC | 09:06 | |
| *** heyongli has joined #openstack-infra | 09:07 | |
| *** dims__ has joined #openstack-infra | 09:08 | |
| jhesketh | yolanda: it appears to be largely taken up by gerrit-ssl-access.log.. maybe the logs aren't being rotated | 09:08 |
| jhesketh | yolanda: how were you checking for rotation? | 09:09 |
| HeOS | Hello! I'm working on the bug: https://bugs.launchpad.net/fuel/+bug/1498389. So, I want to add those persons to 'fuel-plugin-scaleio-cinder-core' group on review.openstack.org, but I have no permission to do it. What I should to have permissions? | 09:09 |
| openstack | Launchpad bug 1498389 in Fuel for OpenStack "Create a Fuel Plugin project in /Openstack" [High,In progress] - Assigned to Andrey Nikitin (heos) | 09:09 |
| yolanda | jhesketh, i checked at /etc/logrotate rules. I checked apache, manage-projects | 09:09 |
| yolanda | but you are right , this one seems missing | 09:09 |
| *** cdent has joined #openstack-infra | 09:10 | |
| yolanda | jhesketh, nope... gerrit-ssl-access is rotated, but today's log is quite big | 09:11 |
| *** dims__ has quit IRC | 09:13 | |
| *** geoffarn_ has quit IRC | 09:13 | |
| rakhmerov | hi, looks like our CI is down now. Are you aware of this? | 09:13 |
| jhesketh | yolanda: the logs are meant to be rotated daily if I'm reading the config correctly, but it looks more like it's been weekly | 09:13 |
| *** geoffarnold has joined #openstack-infra | 09:13 | |
| jhesketh | yolanda: and certainly that log is unusually big | 09:13 |
| ttx | takes forever for zuul to process the results | 09:14 |
| *** links has quit IRC | 09:15 | |
| jhesketh | rakhmerov: in what way | 09:15 |
| rakhmerov | jhesketh: I'm just looking at http://status.openstack.org/zuul/ and filter by "mistral". There's a lot of patches that passed the tests bug keep hanging (for about an hour) and there's also status UNSTABLE for one patch | 09:16 |
| *** heyongli has quit IRC | 09:16 | |
| rakhmerov | which is supposed to be merged | 09:16 |
| ttx | rakhmerov: just takes forever to process results | 09:16 |
| rakhmerov | ttx: just need to wait? | 09:17 |
| *** heyongli has joined #openstack-infra | 09:17 | |
| ttx | rakhmerov: that's what I've been doing for the last few hours :) | 09:17 |
| rakhmerov | ttx: ok, got it. I just wanted to make sure it's being taken care of | 09:18 |
| rakhmerov | thanks | 09:18 |
| yolanda | jhesketh, /var/log/apache2/*.log { | 09:18 |
| yolanda | daily | 09:18 |
| yolanda | and rotate 30 | 09:18 |
| ttx | Given that etherpad, gerrit and zuul are affected, I suspect DoS on the domain | 09:18 |
| *** amotoki has quit IRC | 09:20 | |
| yolanda | ttx, jhesketh, i'm still not used at normal requests to upstream, i can see continuous queries like that | 09:20 |
| yolanda | 50.206.43.50 - - [13/Oct/2015:09:20:09 +0000] "GET /p/openstack/oslo.service/info/refs?service=git-upload-pack HTTP/1.1" 200 22503 "-" "git/1.9.1" | 09:20 |
| yolanda | 50.206.43.50 - - [13/Oct/2015:09:20:10 +0000] "POST /p/openstack/oslo.service/git-upload-pack HTTP/1.1" 200 4196 "-" "git/1.9.1" | 09:20 |
| yolanda | 23.253.216.152 - - [13/Oct/2015:09:20:10 +0000] "GET /p/openstack/requirements/info/refs?service=git-upload-pack HTTP/1.1" 200 291879 "-" "git/1.9.1" | 09:20 |
| yolanda | 104.239.230.132 - - [13/Oct/2015:09:20:10 +0000] "GET /p/openstack/python-zaqarclient/info/refs?service=git-upload-pack HTTP/1.1" 200 52783 "-" "git/1.9.1" | 09:20 |
| yolanda | 50.206.43.50 - - [13/Oct/2015:09:20:10 +0000] "GET /p/openstack/oslo.service/info/refs?service=git-upload-pack HTTP/1.1" 200 22503 "-" "git/1.9.1" | 09:20 |
| yolanda | 172.99.75.81 - - [13/Oct/2015:09:20:10 +0000] "GET /p/openstack/python-openstackclient/info/refs?service=git-upload-pack HTTP/1.1" 200 227334 "-" "git/1.9.1" | 09:20 |
| *** amotoki has joined #openstack-infra | 09:21 | |
| yolanda | they seem legit, but i'm not sure if that load is normal | 09:21 |
| jhesketh | the load and cpu look a little higher than usual | 09:23 |
| yolanda | yes, just checked uptime | 09:23 |
| yolanda | cpu usage is at limits for sure | 09:24 |
| *** amotoki has quit IRC | 09:25 | |
| *** heyongli has quit IRC | 09:27 | |
| *** heyongli has joined #openstack-infra | 09:28 | |
| garyk | is anyone else having issues with gerrit performance? | 09:28 |
| yolanda | garyk, yes, we are working on it | 09:28 |
| garyk | yolanda: ok, thanks | 09:28 |
| jhesketh | yolanda: zuul just got through a chunk of result reporting | 09:28 |
| AJaeger | yolanda, ttx: Time for a topic change or notification to have less people asking? ;) | 09:29 |
| yolanda | jhesketh, so it seems as legit load? | 09:29 |
| jhesketh | there seems to be spike in memory usage at the time of it slowing down | 09:29 |
| jhesketh | yolanda: well nothing is standing out as suspect, but I'm probably missing something | 09:29 |
| jhesketh | I wouldn't say it's normal load anyway | 09:30 |
| loquacities | lots of liberty release tasks, plus summit planning | 09:30 |
| jhesketh | yolanda: ah, we are seeing those jetty issues from before | 09:30 |
| jhesketh | which the apparent fix is to restart gerrit :-s | 09:30 |
| yolanda | jhesketh, on syslog? | 09:30 |
| jhesketh | yolanda: no, in the gerrit error log | 09:30 |
| * jhesketh should have checked sooner | 09:31 | |
| jhesketh | yolanda: perhaps we should bite the bullet and do a gerrit restart | 09:31 |
| ttx | jhesketh: +1 | 09:31 |
| *** nikil89_ has joined #openstack-infra | 09:31 | |
| *** nikil89_ is now known as naggappan | 09:31 | |
| jhesketh | yolanda: I can do the restart if you'd like me to? | 09:32 |
| yolanda | jhesketh, sure | 09:32 |
| yolanda | jhesketh, i thought that these errors were related to high memory usage as well? | 09:32 |
| jhesketh | #status alert gerrit is undergoing an emergency restart to investigate load issues | 09:33 |
| *** heyongli has quit IRC | 09:33 | |
| *** dizquierdo has quit IRC | 09:33 | |
| jhesketh | yolanda: well there is a spike in memory usage | 09:33 |
| jhesketh | hmm, why didn't the statusbot pick up on my message | 09:34 |
| ttx | .whois openstackstatus | 09:34 |
| ttx | it's not around | 09:34 |
| *** geoffarnold has quit IRC | 09:34 | |
| ttx | probably got killed by the same suspect wave | 09:35 |
| jhesketh | maybe... hard to say what's going on | 09:35 |
| *** geoffarnold has joined #openstack-infra | 09:35 | |
| jhesketh | certainly a number of hosts seem affected | 09:35 |
| ttx | jhesketh: would be good to restart that one too :) | 09:35 |
| jhesketh | could be a noisy neighbour for all I know | 09:35 |
| jhesketh | ttx: agreed | 09:35 |
| Zara | hm, storyboard.openstack.org hasn't updated with a change merged on friday; anyone know what's up? | 09:35 |
| *** sergmelikyan has quit IRC | 09:36 | |
| yolanda | etherpad access log shows several spiders accessing it | 09:37 |
| *** glauco_ has joined #openstack-infra | 09:37 | |
| openstackgerrit | gongysh proposed openstack-infra/system-config: my port of ci https://review.openstack.org/234062 | 09:37 |
| jhesketh | I've restarted the statusbot | 09:37 |
| jhesketh | will give it a while to re-join | 09:37 |
| *** openstackstatus has joined #openstack-infra | 09:38 | |
| *** ChanServ sets mode: +v openstackstatus | 09:38 | |
| *** amotoki has joined #openstack-infra | 09:38 | |
| jhesketh | hello openstackstatus | 09:38 |
| * jhesketh waits for it to join the other channels | 09:38 | |
| jhesketh | #status alert gerrit is undergoing an emergency restart to investigate load issues | 09:40 |
| openstackstatus | jhesketh: sending alert | 09:40 |
| jhesketh | thanks openstackstatus | 09:40 |
| *** yfried is now known as yfried|afk | 09:40 | |
| *** yfried|afk is now known as yfried | 09:41 | |
| *** glauco_ has quit IRC | 09:42 | |
| -openstackstatus- NOTICE: gerrit is undergoing an emergency restart to investigate load issues | 09:42 | |
| *** ChanServ changes topic to "gerrit is undergoing an emergency restart to investigate load issues" | 09:42 | |
| jhesketh | yolanda: okay, ready for me to go ahead? | 09:43 |
| yolanda | sure | 09:43 |
| *** haypo has joined #openstack-infra | 09:43 | |
| * jhesketh restarts gerrit | 09:43 | |
| *** sergmelikyan has joined #openstack-infra | 09:43 | |
| openstackstatus | jhesketh: finished sending alert | 09:44 |
| jhesketh | okay service restarted | 09:45 |
| sdague | jhesketh: did you poke the melody data to see what's going on | 09:45 |
| jhesketh | sdague: no, I'm not familiar with melody sorry | 09:46 |
| jhesketh | any hints? | 09:46 |
| jhesketh | oh wait, that's the built in status page right? | 09:46 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Revert "There is no tripleo-incubator package" https://review.openstack.org/233686 | 09:46 |
| sdague | jhesketh: I have no idea, apparently there is some ui in gerrit that shows the garbage collection | 09:46 |
| sdague | clarkb talks about it | 09:46 |
| jhesketh | yeah, that's the one | 09:46 |
| jhesketh | I forgot about it, thanks for the reminder | 09:46 |
| jhesketh | taking a look now | 09:46 |
| sdague | hmmm... maybe we should get clarkb to write a minimanual for gerrit troubleshooting | 09:47 |
| sdague | because this comes up kind of often | 09:47 |
| jhesketh | yeah this problem looks the same to one about a month or so ago where jetty uses up all hte memory | 09:47 |
| jhesketh | apparently the fix is to not use jetty | 09:47 |
| jhesketh | ruagair offered to take a look into it, not sure where he got to | 09:48 |
| *** dizquierdo has joined #openstack-infra | 09:48 | |
| jhesketh | the melody monitoring looks similar to last time | 09:48 |
| sdague | ok, cool | 09:48 |
| openstackgerrit | Merged openstack-infra/project-config: Add openstack/python-scciclient project https://review.openstack.org/218097 | 09:48 |
| *** dims__ has joined #openstack-infra | 09:49 | |
| sdague | the nodepool graph is also... gone | 09:49 |
| jhesketh | yes apparently restarting graphite is needed every now and again... but I've not done that, so I'm not entirely sure that's what is going on | 09:50 |
| *** marzif has quit IRC | 09:50 | |
| * jhesketh checks that now gerrit is doing okayish | 09:50 | |
| sdague | is nodepool actually ok? | 09:50 |
| *** marzif has joined #openstack-infra | 09:51 | |
| openstackgerrit | Thomas Morin proposed openstack-infra/project-config: Test networking-bgpvpn master vs. stable/liberty https://review.openstack.org/233703 | 09:51 |
| *** BharatK_ has quit IRC | 09:52 | |
| yolanda | jhesketh, thanks for taking that | 09:53 |
| jhesketh | was just checking nodepool | 09:53 |
| jhesketh | looks like it's not okay | 09:53 |
| yolanda | also, i think we need to plan some volume resize for logs on gerrit. It's not even arriving to 30 days, and is nearly full | 09:54 |
| *** mrmartin has quit IRC | 09:54 | |
| jhesketh | I suspect it was a change I was baby-sitting earlier https://review.openstack.org/#/c/199750/ | 09:54 |
| jhesketh | yolanda: yep, probably a good idea | 09:54 |
| openstackgerrit | Merged openstack-infra/project-config: make selenium voting in horizon https://review.openstack.org/233744 | 09:55 |
| jhesketh | hmm, maybe it's not due to that change | 09:55 |
| *** mrmartin has joined #openstack-infra | 09:55 | |
| openstackgerrit | Merged openstack-infra/project-config: Add app-catalog-common repository https://review.openstack.org/226999 | 09:55 |
| *** abregman is now known as abregman|lunch | 09:55 | |
| *** geoffarnold has quit IRC | 09:56 | |
| yolanda | jhesketh, what's up with nodepool? in theory these creds are not used now, and nodepool uses secure.conf for that | 09:56 |
| *** geoffarnold has joined #openstack-infra | 09:56 | |
| *** bexelbie has quit IRC | 09:56 | |
| yolanda | jhesketh, i just received a bunch of zuul notifications now | 09:57 |
| *** naohirot has quit IRC | 09:57 | |
| jhesketh | yolanda: right, it's something to do with the new creds http://paste.openstack.org/show/476101/ | 09:57 |
| jhesketh | it looks like it's still launching nodes, so it's possibly working on other clouds | 09:57 |
| *** e0ne has joined #openstack-infra | 09:58 | |
| yolanda | jhesketh, it seems a typo on clouds.yaml | 09:58 |
| yolanda | not related to secure.conf | 09:58 |
| yolanda | checking | 09:59 |
| openstackgerrit | Andreas Jaeger proposed openstack/requirements: Update constraints for openstackdocstheme 1.2.4 https://review.openstack.org/233512 | 09:59 |
| yolanda | weird, rax is on clouds.yaml | 09:59 |
| *** zhenguo has quit IRC | 10:00 | |
| *** mrmartin has quit IRC | 10:00 | |
| *** openstackgerrit has quit IRC | 10:01 | |
| *** openstackgerrit has joined #openstack-infra | 10:02 | |
| *** zeih_ has quit IRC | 10:02 | |
| jhesketh | yolanda: could it have been a nodepool reload pulled in a new nodepool.yaml and didn't update the clouds.yaml? | 10:03 |
| jhesketh | or are they both reloaded | 10:03 |
| yolanda | jhesketh, mm, i cannot tell, because it relies on os_client_config | 10:04 |
| *** BharatK_ has joined #openstack-infra | 10:04 | |
| yolanda | where did you see that, on nodepool logs? | 10:04 |
| jhesketh | yolanda: when running a nodepool command (in that case, nodepool list) | 10:06 |
| jhesketh | output to stdout | 10:06 |
| *** amotoki has quit IRC | 10:06 | |
| yolanda | ah, did you execute with nodepool user? | 10:06 |
| jhesketh | oh right, no | 10:06 |
| yolanda | so, that's one of the side effects of last change to clouds.yaml | 10:06 |
| yolanda | as config for clouds.yaml is on nodepool user | 10:06 |
| jhesketh | yep, that's it | 10:06 |
| yolanda | it needs to be executed with that, otherwise it canot find the creds | 10:07 |
| jhesketh | nodepool looks to be okay in that case | 10:07 |
| yolanda | ++ | 10:07 |
| yolanda | jhesketh, i received a request from OVH to look at https://review.openstack.org/#/c/233699/ | 10:08 |
| yolanda | they suggested to update the rate to 10 seconds per instance, and they wanted to have feedback on it | 10:09 |
| jhesketh | lgtm, shall I +W it? | 10:09 |
| *** bexelbie_ has joined #openstack-infra | 10:09 | |
| yolanda | do we still have max-servers to 0 with ovh? | 10:09 |
| openstackgerrit | Sam Betts proposed openstack-infra/project-config: Enable py3 jobs on networking-cisco https://review.openstack.org/231413 | 10:09 |
| openstackgerrit | Sergey Nikitin proposed openstack/requirements: Updated oslo.db to 3.0.0 https://review.openstack.org/234015 | 10:09 |
| jhesketh | yolanda: yes | 10:10 |
| *** marzif has quit IRC | 10:10 | |
| yolanda | it will be a no op then, i think it's ok to approve and then test it somehow. We could wait for jeblair or fungi to show up, to decide if we can set some servers to ovh... | 10:11 |
| *** marzif has joined #openstack-infra | 10:11 | |
| jhesketh | yolanda: it seems harmless to me, it shouldn't touch their servers until more testing is done | 10:12 |
| yolanda | ok i approve that one, and then let's wait to add some real usage to ovh | 10:13 |
| *** yfried is now known as yfried|afk | 10:13 | |
| jhesketh | sgtm | 10:13 |
| *** pc_m has joined #openstack-infra | 10:14 | |
| jhesketh | yolanda: have you restarted graphite before? | 10:15 |
| yolanda | jhesketh, i think so | 10:15 |
| yolanda | do you want me to restart? | 10:15 |
| jhesketh | yolanda: if you're happy to | 10:15 |
| *** vdrok has joined #openstack-infra | 10:16 | |
| jhesketh | yolanda: well I'm assuming that's what it needs... it doesn't look like anything is reporting stats | 10:17 |
| *** Guest51863 has quit IRC | 10:17 | |
| yolanda | it's an wsgi app, restarting apache should be enough | 10:17 |
| *** geoffarn_ has joined #openstack-infra | 10:17 | |
| *** Sree has joined #openstack-infra | 10:17 | |
| *** dims__ is now known as dims | 10:18 | |
| yolanda | jhesketh, i saw carbon-cache is stopped... do you know about it? | 10:18 |
| jhesketh | nope, no idea sorry | 10:18 |
| *** yamamoto has quit IRC | 10:19 | |
| *** eantyshev has joined #openstack-infra | 10:20 | |
| *** shardy_ has joined #openstack-infra | 10:20 | |
| yolanda | jhesketh, i restarted apache, and started carbon-cache | 10:20 |
| jhesketh | okay | 10:20 |
| jhesketh | yolanda: looks like the nodepool stats are coming in | 10:21 |
| yolanda | great | 10:21 |
| openstackgerrit | Vladimir Kozhukalov proposed openstack-infra/project-config: Create bareon project https://review.openstack.org/231677 | 10:21 |
| *** geoffarnold has quit IRC | 10:21 | |
| *** shardy has quit IRC | 10:22 | |
| *** Sree has quit IRC | 10:22 | |
| jhesketh | yolanda: right, shall I update the status to okay | 10:22 |
| yolanda | jhesketh, silly question, shall i see stats on http://zuul.openstack.org/ ? that's where we see that downstream, but i cannot see any chart on upstream zuul | 10:23 |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack/requirements: Bump oslo.reports version https://review.openstack.org/232204 | 10:23 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: [WIP] Use tripleo.sh to build packages https://review.openstack.org/229425 | 10:23 |
| jhesketh | yolanda: I can't remember if that's connected since we use http://status.openstack.org/zuul/ | 10:23 |
| jhesketh | which shows them | 10:23 |
| openstackgerrit | Andrey Pavlov proposed openstack-infra/project-config: Enabling Heat upgrade for Sahara Grenade job https://review.openstack.org/233061 | 10:24 |
| yolanda | ah, different page there | 10:24 |
| jhesketh | well down the bottom anyway.. the sparks aren't there yet | 10:24 |
| jhesketh | zuul hasn't seemed to send any stats just yet | 10:24 |
| jhesketh | (only nodepool) | 10:24 |
| yolanda | i see a small dot on zuu now | 10:24 |
| yolanda | zuul | 10:24 |
| yolanda | seems to me that is starting | 10:24 |
| yolanda | and no stats on gerrit events | 10:24 |
| jhesketh | yep, just saw that | 10:25 |
| jhesketh | I'd say they are yet to come | 10:25 |
| *** shardy_ has quit IRC | 10:25 | |
| ruagair | I've got no further than our emails jhesketh. | 10:26 |
| *** amotoki has joined #openstack-infra | 10:26 | |
| ruagair | Happy to look into it if it's an ongoing issue. | 10:26 |
| *** hashar has quit IRC | 10:26 | |
| *** shardy has joined #openstack-infra | 10:26 | |
| jhesketh | ruagair: if you have spare cycles that could be helpful :-) | 10:26 |
| *** rdas has joined #openstack-infra | 10:26 | |
| openstackgerrit | Merged openstack-infra/system-config: Set OVH to 10 seconds per request https://review.openstack.org/233699 | 10:27 |
| *** amotoki has quit IRC | 10:28 | |
| rdas | Hi, I have this patch -> https://review.openstack.org/228761 which is failing at the gates as vlan transparency is not enabled on the gates by default. Need help on this | 10:28 |
| *** jlanoux has joined #openstack-infra | 10:29 | |
| *** glauco_ has joined #openstack-infra | 10:31 | |
| *** sshnaidm has quit IRC | 10:35 | |
| *** glauco_ has quit IRC | 10:36 | |
| peristeri | anteaya, are you working on fixing bugs in gerrit? | 10:36 |
| ruagair | OK. I'll add it to my list and take a look on Thursday, jhesketh. | 10:37 |
| jhesketh | thanks ruagair | 10:37 |
| openstackgerrit | Victor Stinner proposed openstack-infra/project-config: Make gate-oslo.rootwrap-python34 voting, add gate https://review.openstack.org/234119 | 10:37 |
| ruagair | Should I scroll back and read it properly? | 10:37 |
| jhesketh | ruagair: I don't think I have any data to add other than what we discussed in email | 10:38 |
| openstackgerrit | Paul Michali proposed openstack-infra/project-config: VPNaaS: Template based functional jobs https://review.openstack.org/232702 | 10:38 |
| *** geoffarn_ has quit IRC | 10:38 | |
| jhesketh | ruagair: basically gerrit getting slow, logs about jetty, high load, restart fixed it | 10:38 |
| yolanda | jhesketh, charts for gerrit are not yet shown | 10:38 |
| *** geoffarnold has joined #openstack-infra | 10:38 | |
| yolanda | and no zuul jobs launched as well | 10:39 |
| jhesketh | yolanda: yeah, I was watching that | 10:39 |
| ruagair | Thanks jhesketh. | 10:39 |
| ruagair | (and yolanda) | 10:39 |
| ruagair | Should we tack this onto the agenda for the morning's meeting? | 10:39 |
| yolanda | i restarted statsd as well | 10:40 |
| jhesketh | ruagair: yep, good idea | 10:41 |
| * ruagair wonders off and adds it. | 10:41 | |
| jhesketh | yolanda: ah right, they are statsd from memory | 10:41 |
| *** ihrachys has joined #openstack-infra | 10:42 | |
| *** kiran-r has quit IRC | 10:43 | |
| *** yfried|afk is now known as yfried | 10:43 | |
| ruagair | "Open Discusison" jhesketh? | 10:43 |
| jhesketh | above that item I'd say | 10:44 |
| ruagair | OK. | 10:44 |
| openstackgerrit | Lucas Alvares Gomes proposed openstack-infra/project-config: Make the ironic-pxe_ssh-dib job to build DIB from source https://review.openstack.org/219351 | 10:45 |
| openstackgerrit | Lucas Alvares Gomes proposed openstack-infra/project-config: Ironic: Create and enable pxe_ipa-ipxe-nv in ironic https://review.openstack.org/233626 | 10:45 |
| *** mpavone has quit IRC | 10:47 | |
| ruagair | There were no tickets, were there jhesketh? | 10:52 |
| openstackgerrit | Dmitriy Ukhlov proposed openstack/requirements: Adds pika RabbitMQ client library https://review.openstack.org/231013 | 10:52 |
| openstackgerrit | Merged openstack/requirements: Bump minimum tempest-lib version https://review.openstack.org/233915 | 10:52 |
| *** mpavone has joined #openstack-infra | 10:52 | |
| *** edmondsw has joined #openstack-infra | 10:52 | |
| *** edmondsw has quit IRC | 10:52 | |
| *** mpavone has quit IRC | 10:54 | |
| *** yfried is now known as yfried|afk | 10:54 | |
| openstackgerrit | gongysh proposed openstack-infra/system-config: my port of ci https://review.openstack.org/234062 | 10:54 |
| ruagair | I've added jhesketh. Please feel free to amend. See you at 05:00 AEST :-) | 10:57 |
| openstackgerrit | Yanis Guenane proposed openstack-infra/puppet-lodgeit: Support LodgeIt installation on EL7 derivatives https://review.openstack.org/234032 | 10:57 |
| jhesketh | ruagair: should be 6am now with DLS | 10:57 |
| tchaypo | Unless Ruagair is in QLD | 10:58 |
| tchaypo | which he is for some part of tomorrow, but I forget which part | 10:58 |
| *** jaypipes has joined #openstack-infra | 10:59 | |
| *** geoffarnold has quit IRC | 10:59 | |
| openstackgerrit | Yanis Guenane proposed openstack-infra/puppet-mysql_backup: Support mysql_backup installation on EL7 derivatives https://review.openstack.org/234029 | 10:59 |
| *** geoffarnold has joined #openstack-infra | 11:00 | |
| openstackgerrit | Gael Chamoulaud proposed openstack-infra/project-config: zuul/layout: Enable puppet check jobs on tripleo-heat-templates. https://review.openstack.org/234129 | 11:00 |
| *** sshnaidm has joined #openstack-infra | 11:00 | |
| *** marzif has quit IRC | 11:01 | |
| *** marzif has joined #openstack-infra | 11:02 | |
| *** jlanoux has quit IRC | 11:04 | |
| ruagair | I'm in QLD for the infra meeting, so yeah, 05:00. | 11:06 |
| ruagair | You're both on AEDT, I'm still on AEST. | 11:06 |
| openstackgerrit | fumihiko kakuma proposed openstack-infra/project-config: Add job for the ovs mechanism driver with native mode https://review.openstack.org/221143 | 11:06 |
| *** yfried|afk is now known as yfried | 11:11 | |
| *** abregman|lunch is now known as abregman | 11:13 | |
| fungi | jhesketh: yolanda: leaving the channel topics changed for a reason? | 11:13 |
| jhesketh | fungi: nope, was waiting for the zuul events to come back, which they have now | 11:13 |
| fungi | looks like gerrit is currently up and working | 11:13 |
| openstackgerrit | Rob Cresswell proposed openstack/requirements: Bump minimum django-babel version https://review.openstack.org/231931 | 11:13 |
| fungi | okay, cool | 11:13 |
| jhesketh | yep, want me to fix the status? | 11:13 |
| yolanda | oh, we have events now | 11:13 |
| openstackgerrit | Rob Cresswell proposed openstack/requirements: Bump minimum Django version https://review.openstack.org/229446 | 11:14 |
| yolanda | jhesketh, so was a matter of time? just to wait for an hour to collect again? | 11:14 |
| jhesketh | yolanda: either that or we hadn't hit the event that does it | 11:14 |
| jhesketh | fungi, yolanda: want me to update the status now? | 11:14 |
| yolanda | i think we are ok yes | 11:14 |
| fungi | usually gerrit restarts are so fast i just use status notice because it'll be back up and running before the alert even finishes updating channel topics | 11:14 |
| *** ociuhandu has quit IRC | 11:14 | |
| jhesketh | heh, okay | 11:15 |
| yolanda | fungi, there were several issues today, not just gerrit. Problems also on etherpad, statusbot, graphite.... | 11:15 |
| jhesketh | #status ok Gerrit has been restarted and is responding to normal load again. | 11:15 |
| openstackstatus | jhesketh: sending ok | 11:15 |
| yolanda | btw, fungi, some important issue on gerrit... only has 3gb left on logs | 11:15 |
| fungi | yolanda: graphite oom'd again? clarkb just restarted carbon-cache yesterday | 11:15 |
| yolanda | fungi, i found it down today, i started | 11:15 |
| AJaeger | yolanda, jhesketh, fungi: AFter the gerrit restart I got lots of emails about merge conflicts. Any idea why this happened? | 11:16 |
| fungi | i have a feeling he only restarted carbon-cache and didn't restart statsd/apache2 yesterday | 11:16 |
| yolanda | i did all restarts | 11:16 |
| fungi | which i've noticed leaves it oom'ing on an almost daily basis | 11:16 |
| fungi | okay, that should help. thanks! | 11:16 |
| *** ChanServ changes topic to "Discussion of OpenStack Developer and Community Infrastructure | docs http://docs.openstack.org/infra/ | bugs https://storyboard.openstack.org/ | source https://git.openstack.org/cgit/openstack-infra/ | channel logs http://eavesdrop.openstack.org/irclogs/%23openstack-infra/" | 11:17 | |
| -openstackstatus- NOTICE: Gerrit has been restarted and is responding to normal load again. | 11:17 | |
| fungi | yeah, looking at http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=32&rra_id=all suggests we should probably clean up some stuff on gerrit. betting we have a lot of junk left sitting around | 11:18 |
| jhesketh | AJaeger: there were a lot of events waiting to be reported... it's possible they are legitimate failures | 11:18 |
| jhesketh | are you able to check? | 11:18 |
| yolanda | fungi, most space is used on logs and backups, it's using logrotate,with 30 days, so we'll need grow disk or shorten policy | 11:19 |
| openstackstatus | jhesketh: finished sending ok | 11:19 |
| AJaeger | jhesketh: most didn't look legitimate, for two that I tried I could rebase manually using gerrit | 11:19 |
| AJaeger | jhesketh: just check the current -1 from Jenkins on project config like https://review.openstack.org/#/c/233909/ - that change was submitted 9 hours ago and shouldn't conflict... | 11:20 |
| jhesketh | it could be a really old report from the failing merger earlier this morning | 11:20 |
| *** geoffarnold has quit IRC | 11:21 | |
| AJaeger | jhesketh: around 50 different changes in my INBOX | 11:21 |
| *** geoffarnold has joined #openstack-infra | 11:21 | |
| fungi | yolanda: yep, i wanted to shorten the log rotation for apache on a lot of our systems to 10 days. just about all our pypi mirrors fill up near instantly because a day of apache logs is larger than the root filesystem | 11:21 |
| AJaeger | jhesketh: might those have been in merge-check queue and the restart confused Zuul? | 11:22 |
| *** devvesa has quit IRC | 11:22 | |
| *** doug-fish has joined #openstack-infra | 11:22 | |
| fungi | gerrit restarts generally don't cause issues for zuul unless you leave gerrit down for an extended period of time | 11:22 |
| AJaeger | jhesketh: just check all these -1 - seldom seen so many: https://review.openstack.org/#/q/status:open+project:openstack-infra/project-config,n,z | 11:22 |
| fungi | if you just `sudo service gerrit restart` it's usually fast enough zuul doesn't notice | 11:23 |
| jhesketh | fungi: that's all it was | 11:23 |
| yolanda | ah fungi, i got some questions from OVH, about updating our rate to 10 in nodepool. We landed that change, but max-servers is set to 0 on nodepool. How do we want to progress on that? | 11:25 |
| *** hashar has joined #openstack-infra | 11:25 | |
| yolanda | jhesketh, we have an issue downstream, our zuul mergers fail quite often with problems finding refs on different branches. Our solution is just to delete the repo in zuul merger, and it's re-created properly next time, but i wonder why do we hit that, is that happening upstream as well? | 11:27 |
| jhesketh | I haven't seen that, no | 11:28 |
| *** marzif has quit IRC | 11:29 | |
| yolanda | jhesketh, we hit that very often: http://paste.openstack.org/show/476113/ | 11:29 |
| *** yfried is now known as yfried|afk | 11:30 | |
| *** jistr is now known as jistr|biab | 11:32 | |
| *** mpavone has joined #openstack-infra | 11:32 | |
| *** mpavone has quit IRC | 11:32 | |
| fungi | yolanda: increase max-servers back to something small like 10 (that's what it was previously) | 11:33 |
| openstackgerrit | yolanda.robla proposed openstack-infra/system-config: Revert "Set ovh max-servers to 0" https://review.openstack.org/234158 | 11:35 |
| yolanda | fungi, i proposed the revert for ovh from 10-0 | 11:35 |
| *** gordc has joined #openstack-infra | 11:36 | |
| fungi | it may merge-conflict on your rate change | 11:36 |
| *** yfried|afk is now known as yfried | 11:37 | |
| *** ldnunes has joined #openstack-infra | 11:37 | |
| yolanda | it didn't fail at same moment, but let's wait for tests | 11:38 |
| *** geoffarnold has quit IRC | 11:42 | |
| *** geoffarnold has joined #openstack-infra | 11:43 | |
| *** thorst has joined #openstack-infra | 11:47 | |
| jhesketh | I'm off, see y'all in the morning | 11:47 |
| *** dims has quit IRC | 11:47 | |
| *** dims has joined #openstack-infra | 11:48 | |
| *** cdoron has joined #openstack-infra | 11:48 | |
| *** yamamoto has joined #openstack-infra | 11:50 | |
| *** kgiusti has joined #openstack-infra | 11:53 | |
| *** otter768 has joined #openstack-infra | 11:53 | |
| *** yamamoto has quit IRC | 11:55 | |
| *** ociuhandu has joined #openstack-infra | 11:56 | |
| *** naohirot has joined #openstack-infra | 11:56 | |
| *** cdoron has quit IRC | 11:57 | |
| naohirot | AJaeger: hello | 11:57 |
| *** amuller has joined #openstack-infra | 11:57 | |
| naohirot | AJaeger: https://github.com/openstack/python-scciclient has been created. | 11:58 |
| naohirot | AJaeger: would you please add me into the two group https://review.openstack.org/#/admin/groups/?filter=python-scciclient ? | 11:58 |
| *** fhubik has joined #openstack-infra | 12:00 | |
| openstackgerrit | Elena Ezhova proposed openstack/requirements: Bump oslo.service version https://review.openstack.org/234026 | 12:00 |
| *** e0ne has quit IRC | 12:00 | |
| *** fhubik is now known as fhubik_brb | 12:00 | |
| naohirot | yolanda: are you there? | 12:01 |
| *** trown|outttypeww is now known as trown | 12:01 | |
| naohirot | AJaeger: yolanda: or someone who can add me into the groups , https://review.openstack.org/#/admin/groups/?filter=python-scciclient | 12:02 |
| *** geoffarnold has quit IRC | 12:03 | |
| *** otter768 has quit IRC | 12:03 | |
| *** fhubik_brb is now known as fhubik | 12:04 | |
| *** geoffarnold has joined #openstack-infra | 12:04 | |
| *** tlian has joined #openstack-infra | 12:04 | |
| *** otter768 has joined #openstack-infra | 12:04 | |
| naohirot | AJaeger: yolanda: or other cores, my name is Naohiro Tamura, naohirot@jp.fujitsu.com | 12:04 |
| *** weshay_xchat has joined #openstack-infra | 12:08 | |
| *** dprince has joined #openstack-infra | 12:08 | |
| yolanda | naohirot, sure | 12:10 |
| naohirot | yolanda: Oh, hi | 12:10 |
| yolanda | btw, i see you as Naohiro Tamura <naohirot.openstack@gmail.com> | 12:10 |
| yolanda | is that ok? | 12:10 |
| naohirot | yolanda: Yes, I use both the gmail and company mail. | 12:11 |
| *** pradk has joined #openstack-infra | 12:11 | |
| naohirot | yolanda: In this case, I prefer the company mail | 12:12 |
| yolanda | ok | 12:12 |
| *** ociuhandu has quit IRC | 12:13 | |
| *** dizquierdo has quit IRC | 12:13 | |
| yolanda | naohirot, so you have one gerrit accoutn with 2 different emails? | 12:13 |
| naohirot | yolanda: Yes, I think so. | 12:14 |
| *** aysyd has joined #openstack-infra | 12:14 | |
| *** salv-orlando has quit IRC | 12:14 | |
| yolanda | i added you , but you show as naohirot.openstack@gmail.com, so that should be your primary email | 12:14 |
| naohirot | yolanda: let me check | 12:14 |
| *** maishsk has joined #openstack-infra | 12:15 | |
| naohirot | yolanda: OK, please add the gmail. | 12:15 |
| yolanda | done | 12:15 |
| naohirot | yolanda: I checked my setting, | 12:15 |
| naohirot | yolanda: the primary mail is gmail, and secondary is comapny | 12:16 |
| yolanda | can you check if you have access now? | 12:16 |
| naohirot | yolanda: Contact Information shows like that. | 12:17 |
| naohirot | yolanda: just a minute | 12:17 |
| *** glauco has joined #openstack-infra | 12:18 | |
| naohirot | yolanda: Umm, scciclient-core lists my gmail, but scciclient-release doesn't. | 12:19 |
| yolanda | ok i'll add you | 12:22 |
| yolanda | done | 12:22 |
| *** rfolco has joined #openstack-infra | 12:22 | |
| naohirot | yolanda: Yeah, thank! both list my gmail. | 12:23 |
| yolanda | cool | 12:23 |
| naohirot | yolanda: In this procedure http://docs.openstack.org/infra/manual/creators.html | 12:24 |
| *** cdoron has joined #openstack-infra | 12:24 | |
| naohirot | yolanda: can I do the rest of things all by myself? | 12:24 |
| *** geoffarnold has quit IRC | 12:24 | |
| *** geoffarnold has joined #openstack-infra | 12:25 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/system-config: Revert "Set ovh max-servers to 0" https://review.openstack.org/234158 | 12:25 |
| naohirot | yolanda: or still is there something I have to ask you? | 12:25 |
| yolanda | you should be ok now | 12:26 |
| naohirot | yolanda: Okay, thanks a lot :) | 12:26 |
| yolanda | no problem | 12:26 |
| *** yamamoto has joined #openstack-infra | 12:28 | |
| *** yamamoto_ has joined #openstack-infra | 12:28 | |
| *** burgerk has joined #openstack-infra | 12:29 | |
| *** ociuhandu has joined #openstack-infra | 12:29 | |
| *** cdoron has quit IRC | 12:30 | |
| HeOS | Hello! I'm working on the bug: https://bugs.launchpad.net/fuel/+bug/1498389. So, I want to add those persons to 'fuel-plugin-scaleio-cinder-core' group on review.openstack.org, but I have no permission to do it. What I should do to have permissions? | 12:30 |
| openstack | Launchpad bug 1498389 in Fuel for OpenStack "Create a Fuel Plugin project in /Openstack" [High,In progress] - Assigned to Andrey Nikitin (heos) | 12:30 |
| *** ociuhandu_ has joined #openstack-infra | 12:31 | |
| *** edmondsw has joined #openstack-infra | 12:31 | |
| *** maishsk has quit IRC | 12:31 | |
| *** achanda has joined #openstack-infra | 12:32 | |
| *** maishsk has joined #openstack-infra | 12:32 | |
| *** yamamoto has quit IRC | 12:33 | |
| *** ociuhandu has quit IRC | 12:34 | |
| *** ociuhandu_ is now known as ociuhandu | 12:34 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-etherpad_lite: Add mysql class to etherpad to optionally configure db https://review.openstack.org/234184 | 12:35 |
| *** glauco___ has joined #openstack-infra | 12:37 | |
| *** achanda has quit IRC | 12:37 | |
| *** glauco has quit IRC | 12:39 | |
| *** tiswanso has joined #openstack-infra | 12:39 | |
| openstackgerrit | Gael Chamoulaud proposed openstack-infra/project-config: zuul/layout: Enable non-voting puppet check jobs on tripleo-heat-templates. https://review.openstack.org/234129 | 12:39 |
| odyssey4me | sdague ping? | 12:40 |
| openstackgerrit | Claudiu Belu proposed openstack/requirements: Adds os-win in projects.txt https://review.openstack.org/233457 | 12:40 |
| *** tiswanso_ has joined #openstack-infra | 12:42 | |
| *** Sree has joined #openstack-infra | 12:42 | |
| yolanda | fungi, about the etherpad change for races, it has +2 from you and me but we both contributed to the change, shall we approve or wait for other +2? | 12:43 |
| openstackgerrit | Gael Chamoulaud proposed openstack-infra/project-config: zuul/layout: Enable non-voting puppet check jobs on tripleo-heat-templates. https://review.openstack.org/234129 | 12:43 |
| *** tiswanso has quit IRC | 12:45 | |
| *** Daisy has joined #openstack-infra | 12:45 | |
| *** e0ne has joined #openstack-infra | 12:46 | |
| *** geoffarnold has quit IRC | 12:46 | |
| *** lawrancejing has joined #openstack-infra | 12:46 | |
| fungi | yolanda: let's see if we can dupe another core reviewer into approving it | 12:46 |
| *** geoffarnold has joined #openstack-infra | 12:46 | |
| openstackgerrit | Naohiro Tamura proposed openstack/requirements: Add python-scciclient into projects.txt https://review.openstack.org/234187 | 12:47 |
| *** sergmelikyan has quit IRC | 12:48 | |
| *** k4n0 has quit IRC | 12:53 | |
| *** rkukura has quit IRC | 12:53 | |
| *** sergmelikyan has joined #openstack-infra | 12:53 | |
| Clint | someone might want to either -t #openstack-meeting or give meetbot ops | 12:54 |
| *** fhubik is now known as fhubik_brb | 12:54 | |
| *** rvba has quit IRC | 12:55 | |
| *** rlandy has joined #openstack-infra | 12:55 | |
| *** Daisy has quit IRC | 12:56 | |
| *** rellerreller has joined #openstack-infra | 12:56 | |
| *** Daisy has joined #openstack-infra | 12:56 | |
| *** rvba has joined #openstack-infra | 12:58 | |
| *** lykinsbd has joined #openstack-infra | 12:58 | |
| *** rvba has quit IRC | 12:58 | |
| *** rvba has joined #openstack-infra | 12:58 | |
| *** Daisy has quit IRC | 13:01 | |
| *** zz_dimtruck is now known as dimtruck | 13:03 | |
| *** jistr|biab is now known as jistr | 13:03 | |
| fungi | grr. silly irc network blips | 13:04 |
| fungi | fixing | 13:04 |
| *** amitgandhinz has joined #openstack-infra | 13:07 | |
| *** geoffarn_ has joined #openstack-infra | 13:08 | |
| *** geoffarnold has quit IRC | 13:08 | |
| *** hdd has joined #openstack-infra | 13:08 | |
| *** Daisy has joined #openstack-infra | 13:08 | |
| *** salv-orlando has joined #openstack-infra | 13:10 | |
| *** achanda has joined #openstack-infra | 13:11 | |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: Add `npm shrinkwrap` to `npm run` macro. https://review.openstack.org/217820 | 13:12 |
| *** baoli has joined #openstack-infra | 13:14 | |
| *** julim has joined #openstack-infra | 13:15 | |
| *** devvesa has joined #openstack-infra | 13:15 | |
| fungi | the current topic control for the openstack bot in #openstack-meeting.* channels relies on chanserv auto-op'ing the bot on join. so if there's a netsplit leaving services partitioned and the bot rejoins on the wrong side of the split, it ends up non-op'ed | 13:15 |
| *** ildikov has joined #openstack-infra | 13:16 | |
| *** Daisy has quit IRC | 13:16 | |
| *** sergmelikyan has quit IRC | 13:16 | |
| fungi | workaround is for an openstack irc admin to `/msg chanserv op #openstack-meeting openstack` (and so on for each of those channels) | 13:16 |
| *** amotoki has joined #openstack-infra | 13:19 | |
| *** achanda has quit IRC | 13:20 | |
| *** alivigni has joined #openstack-infra | 13:20 | |
| *** stevemar_ has joined #openstack-infra | 13:20 | |
| *** MarkAtwood has quit IRC | 13:23 | |
| *** pelix has quit IRC | 13:24 | |
| *** MarkAtwood has joined #openstack-infra | 13:24 | |
| openstackgerrit | Isaac Beckman proposed openstack-infra/devstack-gate: Multinode interfaces on singlenode n-net create once https://review.openstack.org/232513 | 13:24 |
| *** pelix has joined #openstack-infra | 13:25 | |
| *** spzala has joined #openstack-infra | 13:25 | |
| *** nelsnelson has joined #openstack-infra | 13:25 | |
| *** julim has quit IRC | 13:25 | |
| openstackgerrit | Hongbin Lu proposed openstack/requirements: Add certifi to global requirements https://review.openstack.org/221107 | 13:26 |
| *** ihrachys has quit IRC | 13:26 | |
| *** julim has joined #openstack-infra | 13:26 | |
| *** MarkAtwood has quit IRC | 13:27 | |
| jokke_ | hey | 13:27 |
| *** Daisy has joined #openstack-infra | 13:28 | |
| jokke_ | looks like gerrit is acting weird | 13:28 |
| *** ihrachys has joined #openstack-infra | 13:28 | |
| jokke_ | it doesn't seem to clean votes between PSs, like this one https://review.openstack.org/#/c/214342/ | 13:28 |
| *** geoffarn_ has quit IRC | 13:29 | |
| *** geoffarnold has joined #openstack-infra | 13:29 | |
| *** amotoki has quit IRC | 13:29 | |
| openstackgerrit | Isaac Beckman proposed openstack-infra/devstack-gate: Multinode interfaces on singlenode n-net create once https://review.openstack.org/232513 | 13:29 |
| *** jsavak has joined #openstack-infra | 13:29 | |
| *** annegentle has quit IRC | 13:30 | |
| sdague | jokke_: if it detects it as only a rebase, it won't | 13:31 |
| jokke_ | aa, that might have been the case | 13:31 |
| jokke_ | thanks sdague | 13:31 |
| *** burgerk has quit IRC | 13:31 | |
| *** hashar has quit IRC | 13:32 | |
| *** rellerreller has quit IRC | 13:32 | |
| fungi | yep, that's a configurable option we have enabled | 13:32 |
| *** hashar has joined #openstack-infra | 13:32 | |
| openstackgerrit | Naohiro Tamura proposed openstack-infra/project-config: Add check-requirements job for python-scciclient https://review.openstack.org/234211 | 13:32 |
| *** yfried is now known as yfried|afk | 13:33 | |
| fungi | jokke_: it's this feature https://review.openstack.org/Documentation/config-labels.html#label_copyAllScoresOnTrivialRebase | 13:34 |
| *** Daisy has quit IRC | 13:35 | |
| *** Daisy has joined #openstack-infra | 13:36 | |
| *** eharney has joined #openstack-infra | 13:36 | |
| *** yfried|afk is now known as yfried | 13:36 | |
| *** naohirot has quit IRC | 13:38 | |
| openstackgerrit | Sergey Skripnick proposed openstack-infra/project-config: Add new project rally-ci https://review.openstack.org/185365 | 13:38 |
| jokke_ | yeah, sorry, I missed it was rebase | 13:38 |
| *** mestery has joined #openstack-infra | 13:38 | |
| jokke_ | I think I was expecting seeing that rebased comment from gerrit, which doesn't happen if the user has done it locally and uploads | 13:39 |
| *** jsavak has quit IRC | 13:39 | |
| *** amotoki has joined #openstack-infra | 13:40 | |
| *** jsavak has joined #openstack-infra | 13:40 | |
| *** Daisy has quit IRC | 13:40 | |
| krotscheck | mordred: Comments for you on https://review.openstack.org/#/c/232756/ :) | 13:42 |
| *** DericHorn-HP has joined #openstack-infra | 13:42 | |
| ttx | fungi: there are two stuck checks at the top of the queue, we might want to nuke them | 13:43 |
| *** puranamr has joined #openstack-infra | 13:43 | |
| ttx | although... | 13:44 |
| fungi | ttx: top of what queue? | 13:44 |
| ttx | fungi: ignore me | 13:45 |
| fungi | no problem ;) | 13:45 |
| *** puranamr has quit IRC | 13:45 | |
| ttx | fungi: there were two stuck check jobs that appeared not being able to get nodes at top of the queue (while others down seem to be able to). But now it seems it's everyone at the same regime | 13:45 |
| *** adreznec_ is now known as adreznec | 13:46 | |
| sdague | yeh, unit test nodes seem sparse | 13:46 |
| sdague | everyone is waiting on them | 13:46 |
| fungi | i'll make sure there are no stale nodes making nodepool think it doesn't need to boot more | 13:46 |
| *** sergmelikyan has joined #openstack-infra | 13:47 | |
| *** hdd has quit IRC | 13:47 | |
| *** sergmelikyan has quit IRC | 13:48 | |
| fungi | weren't really many (seemed below the usual noise threshold anyway) | 13:48 |
| *** dzamboni has joined #openstack-infra | 13:48 | |
| fungi | but zuul's got a ~2k job backlog at the moment | 13:48 |
| *** geoffarnold has quit IRC | 13:50 | |
| fungi | looks like lots of tempest changes in flight, and a tempest change runs almost 30 jobs in check | 13:50 |
| openstackgerrit | Dolph Mathews proposed openstack/requirements: Add pep257 as a requirement https://review.openstack.org/229685 | 13:50 |
| ttx | sdague: cinder suffers from the webob and the testresources oslo.db thing. I suspect it will need an omnibus fix as well ? If you confirm I can get smcginnis on it | 13:50 |
| *** rkukura has joined #openstack-infra | 13:50 | |
| *** geoffarnold has joined #openstack-infra | 13:50 | |
| openstackgerrit | Cyril Roelandt proposed openstack-infra/project-config: Add a gate-dsvm-functional-py34 job to Neutron https://review.openstack.org/233177 | 13:50 |
| *** sergmelikyan has joined #openstack-infra | 13:51 | |
| openstackgerrit | Dolph Mathews proposed openstack/requirements: Add freezegun to global requirements https://review.openstack.org/228603 | 13:51 |
| *** Shrews has quit IRC | 13:52 | |
| *** Shrews has joined #openstack-infra | 13:52 | |
| *** r-daneel has joined #openstack-infra | 13:55 | |
| *** Piet has quit IRC | 13:56 | |
| *** dizquierdo has joined #openstack-infra | 13:56 | |
| *** ildikov has quit IRC | 13:57 | |
| ttx | fungi: so if you look now, the top three jobs on the check queue do not get test nodes, while the ones below get them | 13:58 |
| clarkb | ttx: that usually means those jobs started and failed due to a jenkins error that zuul detected so it will reschedule | 13:58 |
| clarkb | ttx: common cause there is ssh connection to the slave died | 13:58 |
| *** rdas has quit IRC | 13:59 | |
| ttx | clarkb: ok. I felt like those were stuck this EU morning as well, but that may be twins | 13:59 |
| smcginnis | ttx: pong | 13:59 |
| * smcginnis reads scrollback | 13:59 | |
| clarkb | ttx: it is also possible those changes break the slaves and cause them to loop | 13:59 |
| *** DericHorn-HP has quit IRC | 13:59 | |
| clarkb | though for unittests that is less likely as they don't have sudo | 13:59 |
| ttx | smcginnis: ohai. So we might want to respin a RC to fix the webob + the testrepository issue. In all cases we need a squashed fix that fixes both in one patch | 13:59 |
| ttx | https://review.openstack.org/#/c/233668/ + https://review.openstack.org/#/c/234118/ | 14:00 |
| ttx | smcginnis: I think individually they will fail | 14:00 |
| smcginnis | ttx: I was asking packagers yesterday if that was an issue. | 14:00 |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-etherpad_lite: Add mysql class to etherpad to optionally configure db https://review.openstack.org/234184 | 14:00 |
| smcginnis | Never got a definitive answer. | 14:00 |
| smcginnis | But may be worth getting it in so it doesn't become one. | 14:00 |
| ttx | smcginnis: I think it will be good to have the squashed patch in the pipe in all cases | 14:00 |
| smcginnis | ttx: That sounds like a good idea to me. | 14:00 |
| ttx | so that it's ready to approve if we make a RC3 call | 14:00 |
| ttx | smcginnis: the other two can be abandoned so that we don't generate useless retries | 14:01 |
| *** bgmccollum has left #openstack-infra | 14:01 | |
| *** haypo has left #openstack-infra | 14:01 | |
| smcginnis | ttx: The individual backports? | 14:01 |
| ttx | smcginnis: critical path goes through the check queue right now so the earlier we clear that check the better | 14:01 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 14:01 | |
| ttx | smcginnis: yes, once you propose a squashed version of https://review.openstack.org/#/c/233668/ + https://review.openstack.org/#/c/234118/ we can abandon those two | 14:02 |
| smcginnis | ttx: OK, late for a meeting now, but will work on it. | 14:02 |
| *** tongli has joined #openstack-infra | 14:02 | |
| ttx | thx | 14:02 |
| *** DericHorn-HP has joined #openstack-infra | 14:03 | |
| *** BharatK_ has quit IRC | 14:04 | |
| pabelanger | fungi: yolanda: either of you have time for review / approval for grafyaml this morning? https://review.openstack.org/#/c/232741/ | 14:05 |
| pabelanger | dropping oslo_config dependency | 14:05 |
| *** julim has quit IRC | 14:05 | |
| *** annegentle has joined #openstack-infra | 14:05 | |
| *** rossella_s has quit IRC | 14:06 | |
| *** rossella_s has joined #openstack-infra | 14:07 | |
| *** julim has joined #openstack-infra | 14:07 | |
| *** bswartz has joined #openstack-infra | 14:08 | |
| *** btully has joined #openstack-infra | 14:08 | |
| *** ParsectiX has quit IRC | 14:09 | |
| ttx | smcginnis: I may convince dims to squash the two, since he authored one of them | 14:10 |
| *** MarkAtwood has joined #openstack-infra | 14:10 | |
| *** geoffarnold has quit IRC | 14:11 | |
| *** geoffarnold has joined #openstack-infra | 14:12 | |
| *** mwagner_ has quit IRC | 14:12 | |
| *** burgerk has joined #openstack-infra | 14:13 | |
| annegentle | fungi: is there a way to redirect wiki pages to the various infra manual and contributor guide pages? | 14:14 |
| fungi | annegentle: not automatic redirects, but you can do things like i did for https://wiki.openstack.org/wiki/Vulnerability_Management | 14:15 |
| fungi | basically empty out the article except for the linkable headings, and then turn them into hyperlinks the reader can follow to the equivalent new document/section | 14:16 |
| Zara | hi, asked earlier, but I think it got lost in gerrit-woes: merged change to storyboard on Friday and it's still not showing up; wondering if anyone knows what's going on? | 14:16 |
| Zara | (not showing up on storyboard.openstack.org, that is) | 14:16 |
| *** salv-orlando has quit IRC | 14:17 | |
| annegentle | fungi: ok that's what I was wondering. So there's not an equivalent to 301 redirect in htaccess for mediawiki? | 14:17 |
| dims | e0ne: ping | 14:17 |
| e0ne | dims: hi | 14:17 |
| dims | ttx: let me try delegating to e0ne :) | 14:18 |
| *** hdd has joined #openstack-infra | 14:18 | |
| e0ne | dims: :) | 14:18 |
| e0ne | dims, ttx: does it mean that I need to squosh 2 commits into the one insteas of cherry-pick? | 14:18 |
| dims | e0ne: need your help with squashing https://review.openstack.org/#/c/233668/ + https://review.openstack.org/#/c/234118/ for stable/liberty | 14:18 |
| dims | since you touched one of them already :) | 14:18 |
| * dims is bad with stable branch stuff | 14:18 | |
| ttx | e0ne: yes. One can't merge without the other | 14:18 |
| ttx | e0ne: see https://review.openstack.org/#/c/234166/ for an example | 14:19 |
| ttx | I'll review it | 14:19 |
| dims | e0ne: do you have some time to do that? please let us know | 14:19 |
| e0ne | ttx: thanks! it was my next question what should be in the commit message | 14:19 |
| e0ne | dims: sure, I'll do it asap | 14:19 |
| dims | thanks a ton e0ne | 14:20 |
| fungi | annegentle: not that i'm aware of anyway | 14:20 |
| * dims learning from ttx :) | 14:20 | |
| *** nadya has joined #openstack-infra | 14:21 | |
| *** bdemers has quit IRC | 14:23 | |
| *** bdemers has joined #openstack-infra | 14:24 | |
| *** amotoki has quit IRC | 14:25 | |
| e0ne | dims, ttx: smcginnis just posted update to https://review.openstack.org/#/c/233668/ | 14:26 |
| dims | e0ne: smcginnis: thanks! | 14:26 |
| krotscheck | fungi: I've got a bunch of comments on a governance patch that suggests I put a bunch of things from the Javascript CTI patch to a Javascript Developer's Guide in the infra manual. Do you have any objections? https://review.openstack.org/#/c/232756/ | 14:28 |
| krotscheck | Also, mordred ^^ | 14:28 |
| *** regXboi has joined #openstack-infra | 14:28 | |
| smcginnis | e0ne, ttx: So we can abandon the other review now that this is squashed. Will do that next. | 14:28 |
| ttx | smcginnis: +1 | 14:29 |
| *** yolanda has quit IRC | 14:29 | |
| *** yolanda has joined #openstack-infra | 14:30 | |
| fungi | krotscheck: we've already apparently accumulated a section with python developer suggestions, so it's probably similarly topical... but i think we may want to consider reorganizing it so that the documentation on how to use our infrastructure (the original intent of the infra manual) is distinct from walkthroughs and laundry lists of standard practices | 14:30 |
| *** achanda has joined #openstack-infra | 14:30 | |
| *** flepied has quit IRC | 14:30 | |
| krotscheck | fungi: Or we expand the scope of the infra manual to avoid further doc fragmentation. | 14:31 |
| *** vlaza has quit IRC | 14:31 | |
| fungi | krotscheck: it seems like between the infra manual and the project team guide there's an unfilled gap for a developers reference | 14:31 |
| *** ajmiller has joined #openstack-infra | 14:31 | |
| annegentle | fungi: that's pretty accurate | 14:31 |
| krotscheck | Yep | 14:31 |
| fungi | krotscheck: keeping in mind that the infra team does not need to be making decisions on the best way to declare a variable in a javascript app or how people should name their python decorators | 14:32 |
| annegentle | fungi: I've tried to limit the splitting of many deliverables | 14:32 |
| annegentle | fungi: but, it's mostly about getting core review teams that care about the deliverable | 14:32 |
| krotscheck | An appropriate 'doc project' mission might be "Here's how we build openstack things", and the infra manual actually falls under that scope. | 14:32 |
| alivigni | zxiiro# or zaro Can I get another +2 and this merged please? - https://review.openstack.org/232679 | 14:32 |
| annegentle | so krotscheck do you envision there's a small group of people who would review this content? Would they be mostly involved in the eslint-config-openstack anyway? | 14:32 |
| *** geoffarn_ has joined #openstack-infra | 14:33 | |
| annegentle | krotscheck: I'm leaning towards "get the highest quality reviews you can" | 14:33 |
| *** geoffarnold has quit IRC | 14:33 | |
| *** wenlock has joined #openstack-infra | 14:33 | |
| annegentle | and I would really like for someone to figure out how to build deliverables across multiple repos -- then we get best reviewers and best deliverable collection | 14:33 |
| *** wenlock1 has joined #openstack-infra | 14:34 | |
| krotscheck | annegentle: I'm not certain what you're asking? the eslint project is the javascript equivalent to hacking. There's also openstack-health, ironci-webclient, fuel-ui, horizon, merlin,... basically all the projects that have adopted javascript as a major part of their codebase. | 14:34 |
| *** jsavak has quit IRC | 14:34 | |
| *** salv-orlando has joined #openstack-infra | 14:34 | |
| annegentle | krotscheck: so, who will give you the best reviews? A separate repo or the infra manual or the project guide? | 14:35 |
| krotscheck | annegentle: My effort is to make sure the javascript build ecosystem is as sophisticated as the python ecosystem. | 14:35 |
| krotscheck | annegentle: Depends on the scope of the alternate repo. If it was "Hey here's developer things for openstack", then maybe. If it's just "Here's javascript things", then no. Personally, I'd like to reduce doc fragmentation and piggyback on existing teams. | 14:36 |
| *** jsavak has joined #openstack-infra | 14:36 | |
| annegentle | krotscheck: I think it's better to de-frag as well | 14:36 |
| annegentle | krotscheck: my sense is you'll get more eyeballs in infra and as long as fungi and others are ok with reviewing your patches that's a good place. | 14:37 |
| fungi | i'd love it if the infra manual could be scaled back to just "here's how to interact with the community services and software provided by the infrastructure team" and then split the rest of the content out into one or more separate repos which can have better-focused reviewer groups dedicated to them | 14:37 |
| annegentle | krotscheck: so there's part of your answer :) | 14:37 |
| zxiiro | alivigni: that one's abandoned and it's failing tests | 14:37 |
| alivigni | oops sorry | 14:37 |
| annegentle | krotscheck: the project team guide could have a python section and a javascript section | 14:37 |
| annegentle | rather than starting a new guide | 14:38 |
| alivigni | zxiiro# zaro this one https://review.openstack.org/#/c/232727/ | 14:38 |
| krotscheck | fungi, annegentle: Ok, so it sounds like infra would like to reduce the scope of their responsibilities so they have less things to review and can focus on keeping everything greased :D | 14:38 |
| openstackgerrit | Steve Martinelli proposed openstack/requirements: bump upper constraints of keystonemiddleware and keystoneclient https://review.openstack.org/234255 | 14:38 |
| *** bdemers has quit IRC | 14:38 | |
| krotscheck | I'm all for that. | 14:38 |
| annegentle | krotscheck: yeah that's the balancing act here! | 14:39 |
| krotscheck | annegentle: Is there enough of a core team on the project team guide? | 14:39 |
| annegentle | krotscheck: I can support expanding the project team guide to also "dev guide" | 14:39 |
| fungi | the infra manual ought to be the go-to resource for things like our optimized workflow for pushing changes into the code review system, but the people who curate that information are unlikely to be the same people who curate information on what should be in your commits or what criteria core reviewers would apply when evaluating them | 14:39 |
| annegentle | krotscheck: It's early on, but I'd be happy to support it. I still focus much more on app devs myself than contributor devs. | 14:39 |
| krotscheck | annegentle: Noted. | 14:40 |
| annegentle | (it being adding to the project dev guide) | 14:40 |
| *** MarkAtwood has quit IRC | 14:40 | |
| *** achanda has quit IRC | 14:40 | |
| *** bdemers has joined #openstack-infra | 14:40 | |
| zxiiro | alivigni: done | 14:40 |
| krotscheck | fungi, annegentle: A bit of a nuance question then. The JS governance patch includes specifications like "Infra will call 'npm test' and 'npm run lint'". Should those live in the infra guide, the Javascript CTI governance doc.... | 14:40 |
| alivigni | zxiiro# thanks! | 14:41 |
| annegentle | krotscheck: heh, the reuse of that content is exactly why I'd like to build deliverables across repos | 14:41 |
| fungi | krotscheck: the names of jobs are malleable so i'd be hesitant to encode them in documentation | 14:42 |
| annegentle | krotscheck: try to write it like each rst file is a standalone article/web page | 14:42 |
| *** peristeri has quit IRC | 14:42 | |
| *** xyang1 has joined #openstack-infra | 14:43 | |
| krotscheck | annegentle: When you say "Deliverables across repos"... you mean "This set of docs has this scope, vs. that other one that has a different scope"? | 14:43 |
| fungi | krotscheck: however, from a more general perspective, pretty much all projects have jobs specific to their individual repositories, so documenting them all within the infrastructure team is likely untenable | 14:43 |
| krotscheck | fungi: That's tough, given that we specify tox. | 14:43 |
| annegentle | krotscheck: like this: write an RST file in one repo, publish to docs.openstack.org so that it looks like part of a collection of articles that are related, a guide. | 14:43 |
| krotscheck | annegentle: Oh, gotcha. | 14:44 |
| krotscheck | annegentle: So eslint-config-openstack would have the "Here's our linting rules article", and it'd be collected together into one big book? | 14:44 |
| krotscheck | / site | 14:44 |
| *** amotoki has joined #openstack-infra | 14:44 | |
| annegentle | krotscheck: yeah, with navigation aids | 14:44 |
| krotscheck | fungi: Yeah, that's one of the reasons I want to get ahead of the project diaspora with a good set of docs that people can just copy/paste from | 14:45 |
| annegentle | krotscheck: I know everyone just searches google but still if you land on a page that isn't quite what you were looking for, can you get to something related easily | 14:45 |
| krotscheck | annegentle: Is there an effort to do that projet-based collation? | 14:45 |
| fungi | krotscheck: we (infra) make it easy for projects to use tox to run some jobs, but we provide an open framework for them to implement just about any tools they want (within the constraints of the resources available to us) | 14:45 |
| *** jsavak has quit IRC | 14:45 | |
| fungi | basically all our jobs are just shell scripts with expected return values | 14:45 |
| smcginnis | ttx, others: How would I go about adding to cinder-stable-core? We only have three Cinder folks on there. | 14:45 |
| krotscheck | fungi: Yep. I'm trying to do the same with the npm run macro. "Infra will run lint, you decide what it does" | 14:46 |
| smcginnis | Seems like we should probably have one or two more. | 14:46 |
| smcginnis | I would like to get added, if possible. | 14:46 |
| jgriffith | and smcginnis should probably be one of them :) | 14:46 |
| ttx | smcginnis: I can add names, I just need to make sure they know about the stable branch policy first | 14:46 |
| *** jsavak has joined #openstack-infra | 14:46 | |
| ttx | https://wiki.openstack.org/wiki/StableBranch#Stable_branch_policy | 14:46 |
| smcginnis | ttx: How would you like me to handle that? | 14:46 |
| *** Sree has quit IRC | 14:47 | |
| *** carl_baldwin has joined #openstack-infra | 14:47 | |
| ttx | smcginnis: maybe send an email to me cc all the people you want to add and i'll reply with the pointer and the addition | 14:47 |
| *** geoffarn_ is now known as geoffarnoldX | 14:47 | |
| smcginnis | ttx: Good plan, thanks! | 14:47 |
| *** sergmelikyan has quit IRC | 14:47 | |
| annegentle | krotscheck: I've tried and failed :) | 14:48 |
| *** flepied has joined #openstack-infra | 14:48 | |
| krotscheck | annegentle: Ahhhh | 14:48 |
| annegentle | krotscheck: it would be great if someone else figured it out | 14:48 |
| *** armax has joined #openstack-infra | 14:49 | |
| krotscheck | annegentle: Don't we have awesome docs people on infra that could help? | 14:49 |
| krotscheck | for some reason pleia2 comes to mind, but I think that's just because she's doing i18n things. | 14:49 |
| *** AJaeger has quit IRC | 14:49 | |
| annegentle | krotscheck: it's probably a matter of scripting. | 14:50 |
| *** lawrancejing has quit IRC | 14:50 | |
| annegentle | krotscheck: https://review.openstack.org/#/c/217286/ was my attempt but I would prefer a tagging approach | 14:50 |
| *** zeljkof has joined #openstack-infra | 14:51 | |
| annegentle | krotscheck: I'm trying something similar with an API Guide. It has a /quick-start/ published URL that is sourced in api-site. It has a Compute API collection of RST files sourced in nova | 14:52 |
| annegentle | krotscheck: the nav is kludgy at best confusing at worst | 14:52 |
| *** kgiusti has quit IRC | 14:52 | |
| krotscheck | annegentle: Hrm. Tag based documentation. That wouldn't be horrible. | 14:52 |
| annegentle | krotscheck: with Sphinx you can use directives to "tag" an article (one RST page per article) | 14:53 |
| annegentle | RST file | 14:53 |
| *** arxcruz has quit IRC | 14:53 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Made a common helper function - convert_mapping_to_xml https://review.openstack.org/232727 | 14:54 |
| krotscheck | annegentle: Seems like btween the project tagging for big tent, and this idea, a way of discovering things in openstack by tag could be a global thing. | 14:55 |
| *** jsavak has quit IRC | 14:55 | |
| annegentle | krotscheck: it would be lovely | 14:55 |
| *** bcourt has quit IRC | 14:56 | |
| openstackgerrit | Cyril Roelandt proposed openstack-infra/project-config: Add a gate-dsvm-functional-py34 job to Neutron https://review.openstack.org/233177 | 14:56 |
| *** hashar has quit IRC | 14:56 | |
| *** jsavak has joined #openstack-infra | 14:56 | |
| sdague | ttx: that sounds right | 14:57 |
| *** sergmelikyan has joined #openstack-infra | 14:58 | |
| *** mriedem has joined #openstack-infra | 14:59 | |
| *** maishsk has quit IRC | 14:59 | |
| krotscheck | annegentle: Lemme see if I can get HP to throw a person at that :D (don't hold your breath) | 14:59 |
| *** achanda has joined #openstack-infra | 14:59 | |
| *** sabeen has joined #openstack-infra | 15:00 | |
| *** annegentle has quit IRC | 15:00 | |
| *** tmorin has joined #openstack-infra | 15:01 | |
| *** tmorin has left #openstack-infra | 15:01 | |
| *** dkranz has quit IRC | 15:01 | |
| *** hashar has joined #openstack-infra | 15:02 | |
| *** yfried is now known as yfried|afk | 15:03 | |
| *** achanda has quit IRC | 15:04 | |
| *** abregman has quit IRC | 15:04 | |
| *** angdraug has joined #openstack-infra | 15:05 | |
| pabelanger | pleia2: do you mind reviewing https://review.openstack.org/#/c/233101 | 15:05 |
| tsufiev | hello all! could you tell me please what are the real resources allocated to 'devstack-precise' || 'devstack-trusty' node in many Jenkins jobs? | 15:05 |
| *** salv-orl_ has joined #openstack-infra | 15:05 | |
| clarkb | tsufiev: it depends on the cloud, but 8vcpu and 8GB of RAM with at least 80GB of disk (though these may change with the addition of OVH) | 15:05 |
| sdague | fungi / clarkb any thoughts on why unit test nodes seem starved out? | 15:06 |
| clarkb | tsufiev: is there something you are tring to accomplish? might be better to speak in terms of goals rather than raw resources | 15:06 |
| clarkb | sdague: I can look | 15:06 |
| fungi | sdague: i'll profile the breakdown of node types | 15:06 |
| *** nmagnezi has quit IRC | 15:06 | |
| sdague | we've got this nova stable/liberty fix to make it actually work again which has been waiting for 3 hrs to get any unit test nodes | 15:07 |
| *** salv-orlando has quit IRC | 15:07 | |
| sdague | 234166,1 | 15:07 |
| tsufiev | clarkb, I'm trying to investigate weird Horizon integration test failures (example here http://logs.openstack.org/13/225613/5/check/gate-horizon-dsvm-integration/dbccfbe/), and I'm strongly believe that these failures are caused by Horizon and the rest OpenStack on a devstack node being extremely slow in some aspect | 15:07 |
| *** MarkAtwood has joined #openstack-infra | 15:07 | |
| tsufiev | but to prove this, I need to reproduce this on local devstack with the same resources | 15:08 |
| clarkb | sdague: the gate resets definitely don't help (keystone py34 just reset a chunk of it) | 15:08 |
| fungi | sdague: i see 80 bare-trusty and 250 devstack-trusty nodes in a used state right now | 15:08 |
| clarkb | tsufiev: you should look at the dstat output captured by devstack | 15:08 |
| openstackgerrit | Cyril Roelandt proposed openstack-infra/project-config: Add a gate-dsvm-functional-py34 job to Neutron https://review.openstack.org/233177 | 15:09 |
| sdague | clarkb: sure, but the gate has been short all along | 15:09 |
| clarkb | tsufiev: http://logs.openstack.org/13/225613/5/check/gate-horizon-dsvm-integration/dbccfbe/logs/screen-dstat.txt.gz that is usually a good place to start when you suspect performance issues | 15:09 |
| clarkb | sdague: it gets priority | 15:09 |
| clarkb | and we just hit another reset | 15:10 |
| *** jsavak has quit IRC | 15:10 | |
| sdague | right, but it's only had 3 - 8 jobs all morning | 15:10 |
| *** yfried|afk has quit IRC | 15:10 | |
| *** jsavak has joined #openstack-infra | 15:10 | |
| openstackgerrit | Mikhail S Medvedev proposed openstack-infra/project-config: Add puppet-ciwatch module to openstack-infra https://review.openstack.org/231699 | 15:10 |
| sdague | that's maybe 20 - 30 bare nodes | 15:10 |
| clarkb | but when it resets every 5 minutes that a couple hundred | 15:10 |
| *** alex_xu has quit IRC | 15:10 | |
| tsufiev | clarkb, thanks, will look. By the way, were there some infra changes on October 7? This issue started happening on that date. | 15:11 |
| fungi | now we have 205 devstack-trusty in use and 20 bare-trusty in use | 15:11 |
| *** vmb_out is now known as vmbrasseur | 15:11 | |
| sdague | sure, but if we can't get the fixes in place for those resets, it won't get better :) | 15:12 |
| sdague | fungi: stale nodes? | 15:12 |
| fungi | nope, cleaned those up just a little bit ago and there really weren't any to speak of | 15:12 |
| sdague | ok | 15:12 |
| clarkb | tsufiev: nothing that should cause selenium.common.exceptions.NoSuchElementException: Message: Unable to locate element: {"method":"css selector","selector":"div.modal-dialog"} | 15:13 |
| fungi | because the average run-time for jobs on bare-trusty nodes is so much shorter than the average run-time for jobs using devstack-trusty nodes, that causes us to build more devstack-trusty to satisfy the demand | 15:13 |
| sdague | there was a nodepool restart this morning (as well as many other things) | 15:13 |
| tsufiev | clarkb, okay, will look somewhere else | 15:13 |
| clarkb | tsufiev: that looks to me like your css compiler changed its output or something | 15:14 |
| *** fhubik_brb is now known as fhubik | 15:14 | |
| clarkb | tsufiev: and not a performance issue | 15:14 |
| tsufiev | clarkb, nah, then it would fail in every place, not just in 2 of them | 15:14 |
| sdague | fungi: sure, it looks like 6 bare nodes are available to check right now | 15:14 |
| *** geoffarnoldX has quit IRC | 15:15 | |
| fungi | i think the steady state ends up trying to optimize for a relative per-type ratio of the product of relative run-time and outstanding jobs for that type | 15:15 |
| *** geoffarnold has joined #openstack-infra | 15:15 | |
| sdague | ok | 15:15 |
| tsufiev | clarkb, the thing is that I cannot reproduce this error locally on the fresh devstack | 15:15 |
| fungi | so since we run more devstack-based jobs and they run for longer, the ratio of those tends to skew pretty high | 15:15 |
| *** garyk has quit IRC | 15:15 | |
| openstackgerrit | Kashyap Chamarthy proposed openstack/requirements: global-requirements.txt Bump min. version of oslo.reports https://review.openstack.org/234286 | 15:16 |
| fungi | the goal being to finish the devstack-based jobs and the other faster jobs around the same time | 15:16 |
| clarkb | tsufiev: when you say fresh did you run it on a completel new OS install? | 15:16 |
| clarkb | (we just reset again) | 15:16 |
| sdague | fungi: yeh, seems odd when we have capacity, but so it is | 15:16 |
| tsufiev | clarkb, I reset it to the state where I just installed Ubuntu, but no Devstack yet | 15:17 |
| clarkb | ~3 in 8 minutes | 15:17 |
| jeblair | remember, the allocator now round-robins in order to be "fair" to lesser used node types, so it only allocates bare-trusty (the highest demand) every other pass, so approximately every 10 minutes at the moment. | 15:17 |
| openstackgerrit | Kashyap Chamarthy proposed openstack/requirements: global-requirements.txt: Bump min. version of oslo.reports https://review.openstack.org/234286 | 15:17 |
| fungi | that will definitely contribute to some lag in noticing new demand for less common types too, yes | 15:18 |
| *** bexelbie_ has quit IRC | 15:18 | |
| sdague | jeblair: ok, so 3 hrs is within expected wait time? That's mostly the concern | 15:18 |
| fungi | we're now up to 98 bare-trusty and 203 devstack-trusty in use, so nodepool seems to have noticed the skew again | 15:18 |
| *** Piet has joined #openstack-infra | 15:19 | |
| *** bcourt has joined #openstack-infra | 15:19 | |
| jeblair | sdague: i don't recall saying that. | 15:19 |
| clarkb | tsufiev: does http://logs.openstack.org/13/225613/5/check/gate-horizon-dsvm-integration/dbccfbe/screenshots/test_image_create_delete_2015.10.11-171559.png look correct? | 15:20 |
| clarkb | tsufiev: and is it possible that the virtual frame buffer is affecting things? | 15:20 |
| *** hashar has quit IRC | 15:20 | |
| clarkb | tsufiev: also it uses firefox iirc not chromium | 15:20 |
| jeblair | sdague: i'm just trying to bring pertinent information to the conversation to help analyze the problem. | 15:20 |
| fungi | yeah, i have a feeling that the average job completion on bare-trusty is <10 minutes, which means that the nodes nodepool built for that type are already recycled back into the available quota by the time it realizes it should build more | 15:21 |
| clarkb | tsufiev: also you may want to put the css files in the logs dir so that they can be inspected (at least until this issue is resolved) | 15:21 |
| sdague | fungi: given that a lot are pep8 and docs jobs, that seems reasonable | 15:22 |
| *** annegentle has joined #openstack-infra | 15:22 | |
| tsufiev | clarkb, I've never considerev virtual framebuffer as a weak point :/. The screenshot looks correct, yes. It just meand that although a button was clicked, the modal form that had to appear didn't appear. Thanks for the tip. | 15:22 |
| *** jsavak has quit IRC | 15:23 | |
| tsufiev | might try to change the browser used in tests | 15:23 |
| *** salv-orl_ has quit IRC | 15:23 | |
| *** jsavak has joined #openstack-infra | 15:23 | |
| clarkb | tsufiev: or make a larger framebuffer | 15:24 |
| clarkb | or both | 15:24 |
| jeblair | we're currently allocating 150-250 bare-trusty each pass | 15:24 |
| fungi | that's a substantial proportion of the quota | 15:25 |
| clarkb | (reset) | 15:25 |
| clarkb | so not quite every 5 minutes | 15:25 |
| clarkb | (sorry if I am abusing irc log timestamps) | 15:26 |
| clarkb | and another | 15:26 |
| fungi | wow, yeah, ceilometer and cinder unit tests failing? | 15:27 |
| clarkb | and keystone | 15:27 |
| timrc | Good morning. | 15:27 |
| clarkb | timrc: good morning | 15:27 |
| *** mestery has quit IRC | 15:28 | |
| fungi | but yeah, this constant aborting is not something nodepool's demand calculations are likely to optimize | 15:28 |
| clarkb | fungi: keystone failed quicker than cinder so keystone reset, then cinder reset ahead of keystone, and now keystone will fail again | 15:28 |
| clarkb | in ~4 minutes we will reset again | 15:28 |
| *** salv-orlando has joined #openstack-infra | 15:28 | |
| fungi | if the gate manages to abort a significant proportion of the available quota between each pass on demand calculation, we're really just treading water in the gate pipeline and so starving the check pipeline is expected | 15:29 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Import puppet-grafyaml https://review.openstack.org/234292 | 15:30 |
| fungi | sdague: i guess the core reviewers for affected projects are unaware that they should refrain from approving changes until this situation is resolved? | 15:30 |
| clarkb | btw I checked jenkins masters too to make sure none of them had fallen over and made things worse and they seem happy | 15:30 |
| fungi | same, and also cleaning up what fip cruft i can in hpcloud | 15:31 |
| fungi | just in case | 15:31 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config: Add grafyaml support https://review.openstack.org/233117 | 15:31 |
| fungi | also this sort of rapid node churn has a negative impact on the performance of our providers, leading to longer wait times for replacement nodes in some regions | 15:32 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/project-config: Create tempest-plugin-cookiecutter project https://review.openstack.org/208389 | 15:33 |
| clarkb | reset | 15:33 |
| pabelanger | few reviews up adding puppet-grafyaml into -infra. Will also need governance patch, working on that now | 15:33 |
| sdague | fungi: probably, honestly, I don't know everything that's wrong with the world right now | 15:33 |
| sdague | I was trying to fix the one thing I knew was terrible with liberty | 15:33 |
| sdague | because, you know, release tomorrow :) | 15:34 |
| fungi | hopefully! :/ | 15:34 |
| openstackgerrit | Emma Foley proposed openstack-infra/project-config: Opening a repository for a ceilometer plugin for collectd. https://review.openstack.org/222763 | 15:34 |
| *** cbader_ has joined #openstack-infra | 15:35 | |
| jeblair | ceilometer, cinder, and keystone unit tests are failing? | 15:35 |
| clarkb | jeblair: yes | 15:36 |
| *** geoffarnold has quit IRC | 15:36 | |
| *** salv-orl_ has joined #openstack-infra | 15:36 | |
| jeblair | is there a common cause? | 15:36 |
| fungi | yeah, and comparing the exceptions i see in the job logs for each of them, they look like different (potentially unrelated) failures | 15:36 |
| *** juggler_ has joined #openstack-infra | 15:36 | |
| fungi | not sure if the underlying cause is common between them, but on the surface they're presenting differently | 15:37 |
| *** geoffarnold has joined #openstack-infra | 15:37 | |
| clarkb | ceilometer tests timeout, keystone and cinder have http related problems | 15:37 |
| clarkb | in keystone they try connecting to http:http://example.com which isn't valid | 15:37 |
| pabelanger | fungi: https://review.openstack.org/#/c/234299/ when you have spare cycles. | 15:37 |
| pabelanger | puppet-grafyaml governance patch | 15:38 |
| clarkb | cinder gets a http return code mismatch 400 != 500 | 15:38 |
| *** salv-orlando has quit IRC | 15:38 | |
| jeblair | weird. doesn't seem like the sort of thing that should be able to pass check. | 15:39 |
| jeblair | at least the example.com one | 15:39 |
| mtreinish | sdague, clarkb, fungi, jeblair: if you get a sec can you take a look at: https://review.openstack.org/208389 we've been trying to get that cookiecutter repo started for a while now | 15:39 |
| * clarkb wonders if we should've made one cookiecutter repo with many branches | 15:39 | |
| clarkb | not sure how much overlap they actually have | 15:39 |
| fungi | a fair amount | 15:39 |
| timrc | pabelanger: I'll follow up on your comments in response to my comments. I didn't realize 'session' was private. | 15:39 |
| sdague | mtreinish: later, still trying to fix liberty | 15:39 |
| jeblair | clarkb: it's never too late | 15:39 |
| timrc | I'll have to go take a closer look. | 15:39 |
| *** juggler_ has quit IRC | 15:40 | |
| *** denisra has quit IRC | 15:40 | |
| *** dra_ has joined #openstack-infra | 15:40 | |
| sdague | clarkb: the cinder thing is actually a webob bit I expect | 15:40 |
| mtreinish | sdague: man, you gotta stop living in the past :) | 15:40 |
| sdague | I have no idea about ceilometer fail | 15:40 |
| *** juggler_ has joined #openstack-infra | 15:40 | |
| fungi | yeah, i have a feeling some method in a dependency started prepending http:// to an already fully-formed url or something | 15:40 |
| clarkb | looks like sahara is failing too | 15:40 |
| *** sputnik13 has joined #openstack-infra | 15:41 | |
| clarkb | no module named testresources from oslo_db | 15:41 |
| mtreinish | clarkb: oh, that's probably the extras thing | 15:41 |
| clarkb | which is funny because testr is running the tests | 15:41 |
| fungi | i'm starting to wonder if development freeze for release-managed projects should extend through the entire rc period so that the next cycle gold rush doesn't impede getting the release out the door | 15:41 |
| mtreinish | clarkb: this was a recent change: https://github.com/openstack/oslo.db/blob/master/setup.cfg#L60 which broke a bunch of projects using those fixtures | 15:42 |
| mtreinish | dims: ^^^ that's as much as I know about that one :) | 15:43 |
| clarkb | mtreinish: right but if using testr then testresources should be present | 15:43 |
| clarkb | mtreinish: so it may be a little more complicated than that | 15:43 |
| mtreinish | clarkb: testr doesn't require testresources | 15:43 |
| pabelanger | timrc: Ya, np. Open to other ideas, passing the session object was my first attempt. It is pretty thin right now | 15:43 |
| clarkb | er duh | 15:43 |
| *** juggler_ has quit IRC | 15:43 | |
| clarkb | I am derping and thinking it was testrepository | 15:43 |
| mtreinish | heh, no worries | 15:43 |
| *** maishsk has joined #openstack-infra | 15:44 | |
| *** juggler_ has joined #openstack-infra | 15:44 | |
| clarkb | so I think we need to ask sahara, ceilometer, cinder, and keystone to stop approving changes. Ask them to write fixes for the current issues, Then when they have passed check and are approved let us know so we can promote | 15:45 |
| *** juggler_ has quit IRC | 15:45 | |
| jeblair | we could also consider enqueuing all of their open changes into check | 15:45 |
| jeblair | that would take a while to have an effect though | 15:46 |
| clarkb | SergeyLukjanov: smcginnis gordc stevemar_ ^ please see my message a few lines up | 15:46 |
| *** cdoron has joined #openstack-infra | 15:47 | |
| gordc | clarkb: ack | 15:47 |
| gordc | i think the current failures we have relate to latest oslo.messaging. | 15:48 |
| gordc | will pass it along though. | 15:48 |
| *** asmigala has quit IRC | 15:48 | |
| *** ildikov has joined #openstack-infra | 15:50 | |
| openstackgerrit | Ben Nemec proposed openstack/diskimage-builder: Enable decimal value for $DIB_IMAGE_SIZE. https://review.openstack.org/190026 | 15:51 |
| clarkb | fungi: I agree, we should probably hold off on making a ton of non named release related releases | 15:51 |
| clarkb | fungi: at least until we cut that release | 15:51 |
| openstackgerrit | Caleb Boylan proposed openstack-infra/shade: Make attach_instance return updated volume object https://review.openstack.org/232830 | 15:51 |
| clarkb | fungi: or maybe we should be doing them earlier so that we can fix the issues | 15:51 |
| clarkb | otherwise we run the risk of releasing then breaking it a couple days later | 15:52 |
| *** jlanoux has joined #openstack-infra | 15:52 | |
| *** Swami has joined #openstack-infra | 15:52 | |
| *** harlowja_at_home has joined #openstack-infra | 15:53 | |
| *** maishsk has quit IRC | 15:53 | |
| *** shardy_ has joined #openstack-infra | 15:55 | |
| *** gyee has joined #openstack-infra | 15:55 | |
| *** sshnaidm has quit IRC | 15:56 | |
| *** mestery has joined #openstack-infra | 15:56 | |
| *** shardy has quit IRC | 15:56 | |
| *** e0ne has quit IRC | 15:56 | |
| sdague | clarkb: right, sahara is failing because of the oslo.db testresources thing | 15:57 |
| *** mestery has quit IRC | 15:57 | |
| *** geoffarnold has quit IRC | 15:57 | |
| *** fawadkhaliq has joined #openstack-infra | 15:57 | |
| smcginnis | clarkb: Hold off on merged to master? | 15:58 |
| *** geoffarnold has joined #openstack-infra | 15:58 | |
| smcginnis | s/merged/merges/ | 15:58 |
| elmiko | sdague: was just curious about the sahara failures | 15:58 |
| elmiko | sdague: the bandit gate is non-voting (and was failing before), why the need to remove https://review.openstack.org/233548 ? | 15:59 |
| clarkb | smcginnis: yes | 16:00 |
| *** cdoron has quit IRC | 16:00 | |
| *** shardy_ has quit IRC | 16:00 | |
| sdague | elmiko: it's not bandit that's failing | 16:00 |
| sdague | elmiko: ALL sahara py27 jobs are now falling | 16:00 |
| *** mestery has joined #openstack-infra | 16:00 | |
| sdague | on any patch | 16:00 |
| elmiko | ooph | 16:01 |
| elmiko | anything we can do to help? | 16:01 |
| *** shardy has joined #openstack-infra | 16:01 | |
| fungi | elmiko: not approving more sahara changes until that problem is solved would be a great start | 16:02 |
| elmiko | fungi: ack | 16:02 |
| * elmiko goes back to look through logs | 16:02 | |
| sdague | elmiko: https://review.openstack.org/#/c/232055/ - the fix has been applied to other projects | 16:03 |
| sdague | something like that | 16:03 |
| *** juggler_ has joined #openstack-infra | 16:03 | |
| elmiko | sdague: thanks, i'll take a look | 16:03 |
| *** devkulkarni has joined #openstack-infra | 16:03 | |
| *** _bmjen is now known as bmjen | 16:04 | |
| *** harlowja_at_home has quit IRC | 16:05 | |
| openstackgerrit | Matt Riedemann proposed openstack/requirements: Revert "Temporarily cap webob<1.5.0 in g-r to unblock nova" https://review.openstack.org/233857 | 16:06 |
| timrc | pabelanger: Re: pathing, its just that dashboard is going to be only one component so to having everything under grafana_dashboard seemed odd to me, but that's it. | 16:06 |
| *** yamahata has joined #openstack-infra | 16:06 | |
| *** juggler_ has quit IRC | 16:07 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Adjust jobs after HA Guide RST conversion https://review.openstack.org/234312 | 16:07 |
| *** AJaeger has joined #openstack-infra | 16:07 | |
| *** jlanoux has quit IRC | 16:07 | |
| devkulkarni | hello.. I am looking for some pointers on what to modify in project-config to enable updates to solum's readthedocs site (http://solum.readthedocs.org/en/latest/) .. this site is not updating even after patches updating the docs have been recently merged. anteaya mentioned yesterday to modify solum's docs job in project-config. Currently solum's template has 'python-jobs' and 'openstack-server-publish-jobs'. Are these not the correc | 16:09 |
| *** maishsk has joined #openstack-infra | 16:09 | |
| anteaya | devkulkarni: hi | 16:09 |
| devkulkarni | anteaya: hi | 16:09 |
| *** matrohon has quit IRC | 16:09 | |
| anteaya | were you able to find a project that publishes to readthedocs in project-config? | 16:10 |
| devkulkarni | yes, murano | 16:10 |
| fungi | devkulkarni: your project configuration on rtd is likely incorrect. it should be pulling updates periodically. the hook addition you can add in our infra is just to add a notification to rtd to make it happen sooner than whatever the periodic update schedule is there | 16:10 |
| anteaya | great | 16:10 |
| *** jzerebecki has joined #openstack-infra | 16:10 | |
| jeblair | devkulkarni: solum is an official openstack project, so its canonical documentation is published at http://docs.openstack.org/developer/solum/ | 16:10 |
| jeblair | we usually don't add rtfd hooks for official projects | 16:11 |
| fungi | also that, yes | 16:11 |
| *** _RuiChen_ has joined #openstack-infra | 16:11 | |
| anteaya | jeblair: ah thanks, didn't get around to checking that last night | 16:11 |
| fungi | i forgot solum got their big tent application approved | 16:11 |
| anteaya | so did I | 16:11 |
| devkulkarni | jeblair: thanks!!! that site has updated docs | 16:11 |
| *** flepied has quit IRC | 16:11 | |
| anteaya | devkulkarni: problem solved | 16:11 |
| *** juggler has joined #openstack-infra | 16:11 | |
| elmiko | sdague: any thoughts on mriedem's comment from https://review.openstack.org/#/c/232055 , i'm looking at the other solution and could use a little advice | 16:11 |
| devkulkarni | anteaya: :) | 16:12 |
| anteaya | thanks for following up, good work | 16:12 |
| devkulkarni | fungi: thanks for the info.. probably we can retire rtd then? | 16:12 |
| fungi | devkulkarni: yep | 16:12 |
| mriedem | elmiko: we'd have to revisit that later, oslo.db 3.0 is out now and causing problems | 16:12 |
| devkulkarni | anteaya: :) .. thanks for the help | 16:13 |
| anteaya | devkulkarni: welcome, thanks for asking | 16:13 |
| elmiko | mriedem: have to revisit the solution from 232055? | 16:13 |
| *** jpeeler has quit IRC | 16:13 | |
| *** _RuiChen has quit IRC | 16:14 | |
| *** jpeeler has joined #openstack-infra | 16:14 | |
| *** jpeeler has joined #openstack-infra | 16:14 | |
| mriedem | elmiko: i think so yes | 16:14 |
| mriedem | but right now is kind of a bad time b/c that's impacting stable/liberty release for nova | 16:14 |
| mriedem | see https://review.openstack.org/#/c/234166/ | 16:15 |
| *** MarkAtwo_ has joined #openstack-infra | 16:15 | |
| *** mestery has quit IRC | 16:15 | |
| elmiko | mriedem: ack, thank you. i'll do some reading, might be back with more questions ;) | 16:15 |
| AJaeger | jeblair: we just merged a change for murano to publish for both rtd and docs.openstack.org - on request of the team | 16:16 |
| *** jpeeler has quit IRC | 16:16 | |
| AJaeger | jeblair: should that one have gone in? | 16:16 |
| *** shashank_hegde has joined #openstack-infra | 16:16 | |
| *** jpeeler has joined #openstack-infra | 16:16 | |
| *** MarkAtwood has quit IRC | 16:16 | |
| *** geoffarnold has quit IRC | 16:18 | |
| *** MarkAtwo_ has quit IRC | 16:19 | |
| *** devkulkarni has left #openstack-infra | 16:19 | |
| *** geoffarnold has joined #openstack-infra | 16:19 | |
| jeblair | AJaeger: it's a change from existing practice, at least :) | 16:19 |
| AJaeger | I commented on the change and always wanted to ask whether I should block it or not: https://review.openstack.org/230359 | 16:20 |
| *** unicell has joined #openstack-infra | 16:21 | |
| *** garyk has joined #openstack-infra | 16:21 | |
| AJaeger | So, what is appropriate for future reviews - and what for this change? | 16:21 |
| zaro | morning | 16:21 |
| *** amuller is now known as amuller_afk | 16:22 | |
| anteaya | morning zaro | 16:22 |
| anteaya | AJaeger: I'm following along, I'm interested in the outcome | 16:22 |
| anteaya | and points | 16:22 |
| *** unicell1 has quit IRC | 16:23 | |
| openstackgerrit | Alexey Deryugin proposed openstack-infra/project-config: puppet-murano: enable beaker jobs https://review.openstack.org/234322 | 16:23 |
| *** trown is now known as trown|lunch | 16:23 | |
| openstackgerrit | Merged openstack-infra/system-config: Revert "Set ovh max-servers to 0" https://review.openstack.org/234158 | 16:24 |
| anteaya | so I guess that means our ovh is set to enabled again? | 16:26 |
| anteaya | ovh account | 16:26 |
| Odd_Bloke | Does anyone know of a jenkins-job-builder extension that adds bzr as an SCM option? | 16:27 |
| fungi | anteaya: yep, yolanda was in contact with them about adjusting the api call frequency | 16:27 |
| anteaya | wonderful | 16:28 |
| anteaya | yolanda: thank you | 16:28 |
| yolanda | glad to help | 16:28 |
| clarkb | Odd_Bloke: http://docs.openstack.org/infra/jenkins-job-builder/scm.html doesn't appear to be implemented (yet) | 16:28 |
| *** unicell has quit IRC | 16:28 | |
| elmiko | mriedem, sdague, do i need to get an affects sahara on this bug https://bugs.launchpad.net/nova/+bug/1503501 ? | 16:28 |
| openstack | Launchpad bug 1503501 in neutron "oslo.db no longer requires testresources and testscenarios packages" [High,Fix committed] - Assigned to Davanum Srinivas (DIMS) (dims-v) | 16:28 |
| yolanda | so i shall inform them about us starting tests again | 16:28 |
| Odd_Bloke | clarkb: Yeah, that is regrettably so; I was wondering if anyone knew of a third-party extension that did it. | 16:29 |
| clarkb | Odd_Bloke: you could just add it to JJB :) its usually not hard and iirc the SCM stuff has a lot of reasonable stuff in it to start with | 16:30 |
| *** mestery has joined #openstack-infra | 16:30 | |
| *** rkukura has quit IRC | 16:30 | |
| Odd_Bloke | clarkb: Yeah, I'll look into it. :) | 16:30 |
| *** ifarkas has quit IRC | 16:31 | |
| *** jistr has quit IRC | 16:31 | |
| openstackgerrit | Anne Gentle proposed openstack-infra/project-config: Builds and publishes Compute API Guide https://review.openstack.org/231000 | 16:31 |
| *** dramalho has joined #openstack-infra | 16:33 | |
| *** mestery has quit IRC | 16:33 | |
| *** jsavak has quit IRC | 16:34 | |
| openstackgerrit | Anne Gentle proposed openstack-infra/project-config: Builds and publishes Compute API Guide https://review.openstack.org/231000 | 16:35 |
| bltavares | good evening folks. we have a few acceptance tests for provisioning waiting another +2/+A. Would someone be able to help us with that? | 16:36 |
| *** mestery has joined #openstack-infra | 16:37 | |
| *** jsavak has joined #openstack-infra | 16:37 | |
| AJaeger | jeblair, fungi: Any other comments on my question about murano and RTD/docs.openstack.org, please? | 16:37 |
| *** pvaneck has joined #openstack-infra | 16:37 | |
| zaro | anteaya: you'll be happy to know that the gerrit blocking issue has been fixed. | 16:38 |
| anteaya | zaro: woooohhoooooo | 16:38 |
| anteaya | \o/ | 16:38 |
| anteaya | yes | 16:38 |
| anteaya | you are correct | 16:38 |
| jeblair | AJaeger: i think official projects should favor openstack.org for doc hosting, and we should not have rtfd hooks for them | 16:38 |
| fungi | AJaeger: well, we can't prevent projects from adding themselves from rtd. we _can_ prevent them from adding the on-merge update hook for it, but rtd still updates itself with regularity regardless so doing that has only limited effect but better than nothing | 16:38 |
| anteaya | I am extatic | 16:38 |
| anteaya | zaro: can you share the link to the patch that is the fix? | 16:39 |
| AJaeger | fungi, jeblair : So remove the rtd hooks from zuul/layout.yaml? | 16:39 |
| *** dramalho has quit IRC | 16:39 | |
| *** geoffarn_ has joined #openstack-infra | 16:40 | |
| *** dramalho has joined #openstack-infra | 16:41 | |
| *** dramalho has quit IRC | 16:41 | |
| anteaya | AJaeger: I would +2 that change | 16:41 |
| *** ildikov has quit IRC | 16:41 | |
| zaro | anteaya: https://gerrit-review.googlesource.com/#/c/71380 it was backported to both 2.10 & 2.11 | 16:41 |
| jeblair | it at least prompts a conversation :) | 16:42 |
| jeblair | zaro: yaaaaaaaaaaaaaaaaaaay! | 16:42 |
| *** alexsyip has joined #openstack-infra | 16:42 | |
| openstackgerrit | Merged openstack-infra/shade: Add get_server_by_id optmization https://review.openstack.org/231616 | 16:43 |
| *** dramalho has joined #openstack-infra | 16:43 | |
| *** HeOS has quit IRC | 16:43 | |
| annegentle | AJaeger: yeah, what jeblair and fungi said | 16:43 |
| openstackgerrit | Merged openstack-infra/shade: Accept and emit union of keystone v2/v3 service https://review.openstack.org/233098 | 16:43 |
| greghaynes | I am curious what the argument against also hosting docs on rtd is (in addition to docs.o.o)? | 16:43 |
| *** cbader_ has quit IRC | 16:43 | |
| *** dramalho has quit IRC | 16:43 | |
| jasondotstar | Clint and I were just having a convo about the project renaming effort; thought to continue to convo here... | 16:43 |
| annegentle | greghaynes: seo watering down, people wonder what's correct, official, and so on | 16:43 |
| annegentle | greghaynes: causing confusion instead of alleviating it thru docs | 16:44 |
| anteaya | zaro: nice work, and thank you so much for your persistence on this | 16:44 |
| *** geoffarnold has quit IRC | 16:44 | |
| anteaya | zaro: so where does that put us? | 16:44 |
| greghaynes | We could require a header stating that docs are located at docs.o.o url | 16:44 |
| *** mwagner_ has joined #openstack-infra | 16:44 | |
| anteaya | zaro: are 2.11 and 2.10 options for us for upgrading? | 16:44 |
| zaro | anteaya: that opens the discussion to see which version of Gerrit we want to upgrade too. | 16:44 |
| annegentle | greghaynes: what's the opposite argument, why both? | 16:44 |
| zaro | annegentle: yes | 16:44 |
| annegentle | (just curious, genuinely) | 16:45 |
| anteaya | zaro: with the options being 2.11 and 2.10 | 16:45 |
| * anteaya takes zaro mail to addressed to annegentle for herself | 16:45 | |
| anteaya | annegentle: I think that comment was meant for me | 16:45 |
| zaro | anteaya: yes, i've setup review.o.o with 2.11 and will test it shortly | 16:45 |
| anteaya | zaro: wooooohhhooooooo | 16:45 |
| greghaynes | One thing I have noticed is that indexing for docs.o.o on google is pretty horrible. Currently many of the results are broken and send you to the docs.o.o homepage when they should send you to some other page. Before that issue indexing was just poor in general (I am guessing those two are related) | 16:45 |
| smcginnis | clarkb: Sorry, crazy day today. Can you give me a quick summary why to hold of on master merges and how long we need to do so? | 16:45 |
| anteaya | zaro: you have made my week! | 16:45 |
| greghaynes | This hurts public exposure for projects a lot IMO | 16:46 |
| anteaya | zaro: possibly you have made my whole release | 16:46 |
| anteaya | zaro: you are awesome | 16:46 |
| *** bcourt has quit IRC | 16:46 | |
| *** mestery has quit IRC | 16:46 | |
| annegentle | greghaynes: agreed there's some difficulty with control of the custom search engine and also the sheer breadth of docs.o.o | 16:46 |
| greghaynes | I am going to bet the docs.o.o will always be less good at SEO than rtd, so it is kind of nice to get projects that extra exposure... | 16:47 |
| anteaya | smcginnis: oslo.db doesn't specify its dependencies anymore | 16:47 |
| clarkb | smcginnis: all cinder unittests appear to be failing (this is why why), you need to hold off until cinder fixes the unittests | 16:47 |
| *** notnownikki has quit IRC | 16:47 | |
| annegentle | greghaynes: that's awful and should be fixed then | 16:47 |
| clarkb | anteaya: I don't think cinder was hit by oslo.db | 16:47 |
| jasondotstar | Clint: /me takes a longer look at the script | 16:47 |
| smcginnis | anteaya: We had a patch for an oslo.db issue. | 16:47 |
| anteaya | clarkb: I saw them on the bug: https://bugs.launchpad.net/nova/+bug/1503501 | 16:47 |
| openstack | Launchpad bug 1503501 in Sahara "oslo.db no longer requires testresources and testscenarios packages" [High,In progress] - Assigned to Michael McCune (mimccune) | 16:47 |
| annegentle | greghaynes: perhaps by not publishing so many things to docs.o.o though so there we go :) | 16:47 |
| zaro | annegentle: you might want to try inline edit feature on review-dev.o.o i think that's what you were wanting for a while | 16:47 |
| smcginnis | clarkb: Is that related? | 16:47 |
| annegentle | zaro: ooooooooo! | 16:47 |
| anteaya | smcginnis: sorry I'm confusing things, sorry clarkb | 16:47 |
| clarkb | smcginnis: I don't think oslo.db is the current problem. The current issue was http return code 400 != 500 | 16:47 |
| clarkb | smcginnis: in the unitets | 16:47 |
| jgriffith | clarkb: smcginnis inflight | 16:48 |
| smcginnis | clarkb: Ah, OK. That should be making it's way hough. | 16:48 |
| smcginnis | jgriffith: ;) | 16:48 |
| *** dramalho has joined #openstack-infra | 16:48 | |
| *** sridhar_ram has joined #openstack-infra | 16:48 | |
| jgriffith | https://review.openstack.org/#/c/233923/ | 16:48 |
| *** dramalho has quit IRC | 16:48 | |
| clarkb | jgriffith: smcginnis ok well there are 4 cinder changes in the gate that aren't that change and will all likely fail causing resets | 16:49 |
| clarkb | though one of them passed so maybe not all of them | 16:49 |
| sdague | clarkb: I thought we were caching pip - http://logs.openstack.org/27/209627/6/gate/gate-grenade-dsvm/90da7cd//logs/grenade.sh.txt.gz#_2015-10-13_16_35_24_188 | 16:49 |
| *** jordanP has quit IRC | 16:49 | |
| sdague | clarkb: actually, cinder master changes are fine | 16:49 |
| clarkb | sdague: we are, but I think mtreinish said that grenade doesn't use the cached file | 16:49 |
| clarkb | sdague: they are? | 16:49 |
| *** dramalho has joined #openstack-infra | 16:49 | |
| sdague | yeh, there was one bad 3 way merge change | 16:50 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Do not add rtfd-hooks for murano, rally, sahara https://review.openstack.org/234336 | 16:50 |
| sdague | which I fixed | 16:50 |
| sdague | cinder liberty is all fail until we get their omnibus patch in | 16:50 |
| AJaeger | anteaya, jeblair, fungi, annegentle : Here's my change ^ | 16:50 |
| AJaeger | rally, sahara and murano are involved. | 16:50 |
| jgriffith | clarkb: I got two 232214 232215 | 16:50 |
| sdague | clarkb: at least that's what I've seen of late | 16:50 |
| AJaeger | Should we add PTLs to the review for 234336? | 16:50 |
| *** mestery has joined #openstack-infra | 16:50 | |
| mtreinish | clarkb: well I think we put the cached file somewhere that grenade old doesn't know about | 16:51 |
| mtreinish | I never dug into it though | 16:51 |
| *** dramalho has quit IRC | 16:51 | |
| sdague | mtreinish: ok, maybe stick that fact in the corner and we can look at it once liberty is shipped | 16:51 |
| sdague | clarkb we're compiling a list of liberty must have fixes so we can get them promoted | 16:52 |
| sdague | hopefully soon | 16:52 |
| anteaya | AJaeger: +2 | 16:52 |
| clarkb | ok | 16:52 |
| *** tidwellr1 has left #openstack-infra | 16:52 | |
| clarkb | unrelated http://logs.openstack.org/28/233328/2/check/gate-cinder-python27/02a3371/ has a 44MB console log | 16:53 |
| AJaeger | anteaya: and already a -1 by SergeyLukjanov ;/ | 16:53 |
| anteaya | well you have my opinion | 16:53 |
| sdague | clarkb: oh neat, they are running all unit tests under debug :) | 16:53 |
| AJaeger | yes, thanks, anteaya | 16:53 |
| *** mestery has quit IRC | 16:54 | |
| clarkb | sdague: fwiw jgriffith's fix for cinder is against master | 16:54 |
| sdague | clarkb: yes it was | 16:54 |
| *** fhubik has quit IRC | 16:54 | |
| clarkb | still is | 16:54 |
| mtreinish | clarkb: do they still have the crazy osprofiler stack trace on success that repeats the same call a bunch of times? | 16:54 |
| sdague | but it 3 way merged in a funny way to pass check, and go into gate, but fail there | 16:54 |
| clarkb | mtreinish: yes I think that may be what is happening | 16:54 |
| *** davidlenwell has quit IRC | 16:54 | |
| sdague | I fixed it | 16:55 |
| anteaya | AJaeger: welcome | 16:55 |
| sdague | unless we are talking about another patch | 16:55 |
| clarkb | sdague: ok so cinder does need to hold off on master then right? | 16:55 |
| *** luqas has joined #openstack-infra | 16:55 | |
| sdague | clarkb: not as far as I know | 16:55 |
| clarkb | sdague: https://review.openstack.org/#/c/233923/ needs to merge | 16:55 |
| clarkb | without it master fails | 16:55 |
| sdague | clarkb: no, master should pass without it | 16:55 |
| SergeyLukjanov | AJaeger, hey, sorry about this - in Sahara we're using rtfd for have stable/* branches doc snapshots for users (were requested many times when people using sahara and it's not the master and they want to have docs for the corresponding version) | 16:55 |
| Clint | jasondotstar: cool | 16:55 |
| *** davidlenwell has joined #openstack-infra | 16:56 | |
| sdague | clarkb: the issue was that v1 of that patch only changed the code, and not the test | 16:56 |
| sdague | and some how magic merged to a success in check | 16:56 |
| *** scheuran has quit IRC | 16:56 | |
| *** dramalho has joined #openstack-infra | 16:56 | |
| sdague | but then was failing because the test assert was 400 != 500 | 16:57 |
| sdague | my update is 500 == 500 | 16:57 |
| sdague | master is 400 == 400 | 16:57 |
| jeblair | OH! that's why i'm getting over quota messages in ovh | 16:57 |
| clarkb | sdague: so what fixes the 400 != 500? | 16:57 |
| *** cdoron has joined #openstack-infra | 16:57 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Revert "Revert "Set ovh max-servers to 0"" https://review.openstack.org/234341 | 16:57 |
| clarkb | I thought it was 233923 | 16:57 |
| *** cdent has quit IRC | 16:57 | |
| jeblair | yolanda, fungi: ^ we are not at all ready to use ovh in production | 16:57 |
| sdague | v1 of 233923 introduced 400 != 500 | 16:58 |
| sdague | that was the cause of it | 16:58 |
| jeblair | yolanda, fungi: ^ i'm still trying to run tests locally | 16:58 |
| jeblair | (and have spent all morning wondering why i can't get enough servers to run a test) | 16:58 |
| *** amuller_afk is now known as amuller | 16:59 | |
| sdague | cinder kilo seems borked though | 16:59 |
| *** ivar-lazzaro has joined #openstack-infra | 16:59 | |
| yolanda | jeblair, ok, i wasn't aware. Can you update OVH people with test status? they asked for feedback | 16:59 |
| *** bcourt has joined #openstack-infra | 16:59 | |
| jeblair | yolanda: i have nothing to tell them yet, sorry. | 16:59 |
| *** dangers_away has joined #openstack-infra | 17:00 | |
| *** DericHorn-HP has quit IRC | 17:00 | |
| jeblair | yolanda: i hope to by the end of today | 17:00 |
| yolanda | jeblair ok | 17:00 |
| *** skylerberg has joined #openstack-infra | 17:00 | |
| jeblair | yolanda: but it takes many hours to set up a devstack run, which may fail. it's what i spent all day yesterday on. | 17:01 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Experimental fedora-22 node for puppet-apply tests https://review.openstack.org/231679 | 17:01 |
| *** flepied has joined #openstack-infra | 17:01 | |
| yolanda | jeblair is there any way i can help? what's blocker for that? | 17:02 |
| jeblair | yolanda: it's just really, really slow. | 17:02 |
| *** unicell has joined #openstack-infra | 17:02 | |
| jeblair | yolanda: i started trying to spin up 10 instances 2 hours ago, and have not succeeded yet. | 17:03 |
| jeblair | yolanda: i spent 30 minutes finding a devstack run to mimic and configuring it. | 17:03 |
| jeblair | yolanda: i left 30 instances running multinode tests last night, and they all failed during the setup phase for some unknown reason | 17:03 |
| yolanda | jeblair, so not ready for production for sure | 17:04 |
| AJaeger | SergeyLukjanov: did you check http://docs.openstack.org/developer/sahara/kilo/ ? | 17:04 |
| *** doug-fis_ has joined #openstack-infra | 17:04 | |
| AJaeger | SergeyLukjanov: see also the discussion above, we should have one reference for our documentation... | 17:04 |
| SergeyLukjanov | AJaeger, for how long time it's available? | 17:04 |
| *** doug-fis_ is now known as doug-fish_ | 17:05 | |
| jeblair | yolanda: there's at least 2 reasons it's not ready for production -- we can't run at a 10 second rate -- nodepool will try to allocate all of its nodes from ovh and just stop. but also, we think that ovh nodes are slower, which is what i'm trying to quantify. | 17:05 |
| AJaeger | SergeyLukjanov: I don't think we ever delete automatically on docs.o.o | 17:05 |
| AJaeger | SergeyLukjanov: http://docs.openstack.org/developer/sahara/icehouse/ is there as well | 17:05 |
| *** maishsk has quit IRC | 17:06 | |
| jeblair | yolanda: but i want to go back to them with "we know these nodes are slower by x%, let's change the flavor" and also "yes 10 seconds works, but we need to hit 0.1 in order to use it in production" | 17:06 |
| fungi | jeblair: mordred: when you have a sec can you look at cleaning up unused files in your homedirs on review.o.o? should be just enough for me to force rotation of the apache access log there which has been failing due to insufficient space to copy the file for the past few weeks | 17:06 |
| *** zz_gondoi is now known as gondoi | 17:06 | |
| *** skolekonov is now known as _skolekonov | 17:06 | |
| pabelanger | timrc: Ya, I mostly got the structure from JJB. We can modify it as needed | 17:06 |
| *** doug-fish has quit IRC | 17:07 | |
| anteaya | fungi: ah thank you for tracking that down | 17:07 |
| anteaya | fungi: as of right now how much space does it need to rotate the logs? | 17:07 |
| *** vkmc is now known as vkmc-afk | 17:08 | |
| jeblair | fungi: all gone | 17:08 |
| jeblair | fungi: somehow my home directory had a transporter accident with a git repo :/ | 17:08 |
| anteaya | jeblair: :( | 17:09 |
| anteaya | like galaxy quest | 17:09 |
| *** jsavak has quit IRC | 17:09 | |
| anteaya | and the chicken lizard | 17:09 |
| *** jsavak has joined #openstack-infra | 17:10 | |
| fungi | jeblair: another gigabyte should be sufficient | 17:10 |
| openstackgerrit | Hamdi Roumani proposed openstack-infra/project-config: Add #storlets to gerritbot https://review.openstack.org/234350 | 17:10 |
| *** devvesa has quit IRC | 17:11 | |
| *** naggappan has quit IRC | 17:11 | |
| *** maishsk has joined #openstack-infra | 17:11 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config: Bump puppet-memcached to v2.6.0 https://review.openstack.org/192741 | 17:12 |
| sdague | clarkb: ok - https://etherpad.openstack.org/p/liberty-release-gate-race | 17:12 |
| sdague | there are 2 patches at the top for liberty, which are cinder and nova fixes | 17:13 |
| *** carl_baldwin has quit IRC | 17:13 | |
| sdague | that we need as soon as we can for rc3s | 17:13 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config: Bump puppet-memcached to v2.6.0 https://review.openstack.org/192741 | 17:13 |
| sdague | so if we could promote those 2 in the gate queue, that would be swell | 17:13 |
| sdague | the nova one is already in queue | 17:13 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config: Create stackalytics.o.o under -infra https://review.openstack.org/190765 | 17:13 |
| sdague | the cinder one is in check waiting for nodes, but we're confident on it | 17:14 |
| jgriffith | sdague: adding this one to mater for cinder https://review.openstack.org/#/c/233923/ | 17:14 |
| sdague | jgriffith: that's a master fix though | 17:14 |
| *** achanda has joined #openstack-infra | 17:14 | |
| *** gordc has quit IRC | 17:14 | |
| sdague | right now, just worrying about liberty | 17:14 |
| sdague | because, you know, release | 17:14 |
| jgriffith | sdague: sorry... meant to your etherpad | 17:14 |
| sdague | oh, sure, but I still don't understand why that's needed to pass tests | 17:14 |
| anteaya | pabelanger: so this one has a comment from Aug 5th: https://review.openstack.org/#/c/190708/10 | 17:15 |
| *** maishsk has quit IRC | 17:15 | |
| sdague | I think 233923 can land the normal way | 17:15 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Ensure we actually purge all puppet related directories https://review.openstack.org/208088 | 17:15 |
| sdague | I guess clarkb is no longer around, fungi / jeblair either of you want to help with the promote? | 17:16 |
| *** geoffarn_ has quit IRC | 17:16 | |
| *** nmagnezi has joined #openstack-infra | 17:16 | |
| fungi | sdague: yep, i'll take care of it | 17:16 |
| sdague | https://review.openstack.org/#/c/234166/ and https://review.openstack.org/#/c/234329 | 17:17 |
| jgriffith | sdague: yeah, good point; ok nm... let me go poke around to see what was being reported earlier | 17:17 |
| *** ivar-lazzaro has quit IRC | 17:17 | |
| *** geoffarnold has joined #openstack-infra | 17:18 | |
| sdague | jgriffith / smcginnis - someone should look into the kilo across the board fail though | 17:19 |
| *** MarkAtwood has joined #openstack-infra | 17:19 | |
| pabelanger | anteaya: Yup, still working through backlog. need to get development enviroment going again for stackalytics | 17:19 |
| anteaya | pabelanger: okey dokey | 17:19 |
| anteaya | pabelanger: thank you for working on this | 17:19 |
| sdague | jgriffith / smcginnis - http://logs.openstack.org/45/232845/2/gate/gate-cinder-python27/3c93611/console.html#_2015-10-13_17_08_26_575 | 17:20 |
| *** asselin_ has joined #openstack-infra | 17:20 | |
| *** mestery has joined #openstack-infra | 17:20 | |
| smcginnis | sdague: pbr issue? I hadn't seen that. | 17:21 |
| *** mtanino has joined #openstack-infra | 17:21 | |
| sdague | I think there is some reference file you need to update post kilo release, which has not happened yet | 17:21 |
| smcginnis | sdague: OK, looking in to it. | 17:22 |
| *** asselin__ has quit IRC | 17:22 | |
| *** nadya has quit IRC | 17:23 | |
| *** markus_z has quit IRC | 17:23 | |
| smcginnis | sdague: Sorry, not too familiar with this one. | 17:23 |
| smcginnis | sdague: Does this: https://github.com/openstack/cinder/blob/stable/kilo/setup.py#L28 | 17:23 |
| smcginnis | sdague: Need this: https://github.com/openstack/cinder/blob/master/setup.py#L28 | 17:24 |
| *** salv-orl_ has quit IRC | 17:24 | |
| sdague | smcginnis: I believe that since you released 2015.1.2 you must now bump https://github.com/openstack/cinder/blob/1ec74b88ac5438c5eba09d64759445574aa97e72/setup.cfg#L3 | 17:25 |
| *** salv-orlando has joined #openstack-infra | 17:25 | |
| sdague | or maybe this is stable team job | 17:25 |
| openstackgerrit | Andreas Jaeger proposed openstack/requirements: Move openstackdocsstheme to blacklist https://review.openstack.org/234357 | 17:26 |
| smcginnis | sdague: Looks like it was a stable team thing in the past: | 17:26 |
| smcginnis | https://github.com/openstack/cinder/commits/1ec74b88ac5438c5eba09d64759445574aa97e72/setup.cfg | 17:26 |
| smcginnis | sdague: But I can take care of it right now. | 17:26 |
| mtreinish | smcginnis: I would just drop that line, let pbr handle that automagically from the tags | 17:27 |
| smcginnis | mtreinish: IIRC, the version was added there for a reason. | 17:27 |
| sdague | mtreinish: is that the proscribed method? | 17:27 |
| sdague | honestly, I don't know, expect that it explodes | 17:27 |
| pabelanger | fungi: do I need to update our stackalytics spec to add a stackalytics-dev.o.o host? | 17:27 |
| openstackgerrit | Clint Adams proposed openstack-infra/project-config: Add script to facilitate project renaming in {projects,channels,layout}.yaml https://review.openstack.org/228002 | 17:28 |
| mtreinish | smcginnis: I don't know why they'd add it | 17:28 |
| mtreinish | sdague: that's what I've always been told to do whenever I hit that pbr semver issue on a project after a release | 17:28 |
| fungi | pabelanger: not necessarily. i have a feeling at least initially stackalytics.o.o will be the dev host until stackalytics.o is retired/cut over | 17:28 |
| mtreinish | smcginnis: until a recent-ish pbr version it wouldn't fail on the version line being < the latest tag | 17:29 |
| pabelanger | fungi: okay, ya. if you are fine with using that. I'll update some code | 17:29 |
| sdague | mtreinish: it's supposed to be bumped | 17:29 |
| sdague | that's the advice in the relmgr office | 17:30 |
| smcginnis | mtreinish: It appears that was updated from a global requirement push. | 17:30 |
| *** ybathia has joined #openstack-infra | 17:31 | |
| *** achanda has quit IRC | 17:31 | |
| *** claudiub has quit IRC | 17:33 | |
| *** chair6 has joined #openstack-infra | 17:33 | |
| openstackgerrit | Merged openstack-infra/ciwatch: Add CI Watch, a third-party CI monitoring dashboard https://review.openstack.org/231595 | 17:34 |
| *** Hal has joined #openstack-infra | 17:34 | |
| openstackgerrit | Merged openstack-infra/ciwatch: Allow external access by default https://review.openstack.org/231596 | 17:35 |
| *** mestery has quit IRC | 17:35 | |
| *** chair6 has left #openstack-infra | 17:36 | |
| *** HeOS has joined #openstack-infra | 17:37 | |
| *** geoffarnold has quit IRC | 17:39 | |
| openstackgerrit | Merged openstack-infra/ciwatch: Allow global ciwatch conf file https://review.openstack.org/231597 | 17:39 |
| openstackgerrit | Merged openstack-infra/ciwatch: Fix sqlalchemy models so they work with MySQL backend https://review.openstack.org/231598 | 17:39 |
| *** geoffarnold has joined #openstack-infra | 17:39 | |
| *** nikhil has quit IRC | 17:39 | |
| mordred | fungi: done | 17:40 |
| *** nikhil has joined #openstack-infra | 17:40 | |
| anteaya | fungi: I'd like to add zaro's gerrit announcement to today's meeting agenda and perhaps discuss where that puts us for a gerrit upgrade | 17:41 |
| anteaya | fungi: where in the agenda would you like me to put it? | 17:41 |
| fungi | anteaya: stick it in before open discussion and i'll make sure to reserve some time for it | 17:41 |
| *** ihrachys has quit IRC | 17:41 | |
| fungi | mordred: thanks! | 17:41 |
| *** gordc has joined #openstack-infra | 17:41 | |
| openstackgerrit | Merged openstack-infra/ciwatch: Remove hardcoded data path https://review.openstack.org/231599 | 17:41 |
| openstackgerrit | Merged openstack-infra/ciwatch: Move configuration step before installation https://review.openstack.org/231600 | 17:41 |
| anteaya | fungi: okey dokey, thank you | 17:42 |
| *** hdd has quit IRC | 17:42 | |
| *** xyang1 has quit IRC | 17:42 | |
| *** annegentle has quit IRC | 17:42 | |
| anteaya | zaro: so will you be around for the infra meeting today and you can discuss the significance of https://gerrit-review.googlesource.com/#/c/71380/2 at the meeting? | 17:43 |
| *** trown|lunch is now known as trown | 17:43 | |
| zaro | anteaya: sure can. | 17:44 |
| anteaya | zaro: thank you | 17:44 |
| anteaya | yay | 17:44 |
| * anteaya continues to do a happy dance | 17:45 | |
| *** vkmc-afk is now known as vkmc | 17:45 | |
| * anteaya showers zaro with praise and the smiles of angels | 17:46 | |
| * zaro sees future for anteaya on dancing with the stars | 17:46 | |
| anteaya | no | 17:46 |
| anteaya | I have sworn off high heels | 17:47 |
| anteaya | thanks though :) | 17:47 |
| openstackgerrit | Daniel Watkins proposed openstack-infra/jenkins-job-builder: Implement bzr SCM module. https://review.openstack.org/234365 | 17:47 |
| *** annegentle has joined #openstack-infra | 17:48 | |
| zaro | with tevas of course :) | 17:48 |
| *** achanda has joined #openstack-infra | 17:48 | |
| Odd_Bloke | clarkb: ^ your wish is my command (this time :p). | 17:48 |
| *** yfried|afk has joined #openstack-infra | 17:49 | |
| anteaya | zaro: of course :) | 17:52 |
| *** nadya has joined #openstack-infra | 17:52 | |
| *** hdd has joined #openstack-infra | 17:54 | |
| clarkb | notmyname: I was chatting with greghaynes about the swift mod time thing. How feasible would it be to have swift mimic the posix behavior of updating the timestamp when something one level below it updated by POSTing to an object whose name was the hash of the path. Then when pulling the object back out again swift could get the mod time from the hashed name object and use that. Would probably need | 17:54 |
| clarkb | to be a per container setting off by default that accepts the delimiter you wish to use | 17:54 |
| openstackgerrit | John L. Villalovos proposed stackforge/gertty: Use 'R' for reverse sort instead of 'r' https://review.openstack.org/234369 | 17:54 |
| *** esmute__ has joined #openstack-infra | 17:55 | |
| fungi | yolanda: interesting. after trying to manually run logrotate i discovered that our /etc/logrotate.d/apache2 config has syntax errors. digging deeper now but that may have contributed to the lack of rotation (i couldn't get a manual run to work until i fixed a couple of doubled backslashes) | 17:55 |
| *** abitha has joined #openstack-infra | 17:55 | |
| *** sergmelikyan has quit IRC | 17:56 | |
| *** serg_melikyan has joined #openstack-infra | 17:56 | |
| *** serg_melikyan has quit IRC | 17:56 | |
| *** sergmelikyan has joined #openstack-infra | 17:57 | |
| *** nadya has quit IRC | 17:57 | |
| reed | mordred, calebb and I are venturing in trying shade with ceph ... wish us luck :) | 17:57 |
| fungi | yolanda: aha, yep, the error was introduced by https://review.openstack.org/206582 | 17:57 |
| fungi | fix on the way | 17:57 |
| mordred | reed: good luck!!! | 17:57 |
| *** dprince has quit IRC | 17:58 | |
| mordred | reed: I look forward to your findings | 17:58 |
| *** fawadkhaliq has quit IRC | 17:58 | |
| *** asselin_ is now known as asselin__ | 17:58 | |
| *** sergmelikyan has quit IRC | 17:59 | |
| *** sabeen has quit IRC | 17:59 | |
| *** sergmelikyan has joined #openstack-infra | 17:59 | |
| yolanda | fungi, is that for postrotate and prerotate? manifest was applied correctly and logrotate rules are on place | 17:59 |
| *** MarkAtwood has quit IRC | 17:59 | |
| fungi | yolanda: yep | 17:59 |
| reed | we're stuck at the start line: auth :) | 17:59 |
| notmyname | clarkb: let me think on that | 17:59 |
| *** armax has quit IRC | 17:59 | |
| fungi | yolanda: it looks like 'fi; \\' is causing \\ to wind up on the end of the line | 17:59 |
| fungi | yolanda: "stuff \\" just ends up with \ as intended | 18:00 |
| *** Sukhdev has joined #openstack-infra | 18:00 | |
| *** nadya has joined #openstack-infra | 18:00 | |
| *** geoffarnold has quit IRC | 18:00 | |
| *** geoffarnold has joined #openstack-infra | 18:00 | |
| fungi | yolanda: i have a feeling this may be something puppet lint will need a workaround for | 18:00 |
| *** MarkAtwood has joined #openstack-infra | 18:00 | |
| yolanda | fungi, yes, totally missed that | 18:01 |
| yolanda | btw, in gerrit this had not affected at least, because we are not even reaching 30 days | 18:01 |
| yolanda | it could have affected in short days | 18:01 |
| *** annegentle has quit IRC | 18:01 | |
| fungi | yolanda: well, 30 days is also way too long for rotating that on gerrit. the apache log grew to nearly 7gb over the past few weeks | 18:02 |
| yolanda | yes, do you think we shall shorten that generally on apache? or have a lower policy on some instances like gerrit, and default to 30 generally? | 18:03 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-httpd: Clean up doubled backslashes in logrotate config https://review.openstack.org/234373 | 18:03 |
| fungi | yolanda: i think for gerrit and pypi mirrors we can just shorten it to daily rotation with 10 days retention or something | 18:03 |
| *** maishsk has joined #openstack-infra | 18:04 | |
| yolanda | i don't think that we have this logrotate class applied generally yet | 18:04 |
| *** Sukhdev has quit IRC | 18:04 | |
| fungi | there's a fix, though no idea if puppet lint will complain | 18:04 |
| yolanda | just to gerrit from what i know | 18:04 |
| fungi | yolanda: thanks. the pypi mirrors would be the next priority i think, since one day's worth of apache logs is enough to use up all available space in / for some of them | 18:05 |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-etherpad_lite: Add mysql class to etherpad to optionally configure db https://review.openstack.org/234184 | 18:07 |
| * clarkb is excited for the release to happen so that he can clean up the devstack-gate PROJECTS list without causing too much trouble | 18:07 | |
| *** kgiusti has joined #openstack-infra | 18:07 | |
| *** nadya has quit IRC | 18:07 | |
| *** fawadkhaliq has joined #openstack-infra | 18:07 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-gerrit: Shorten logrotate policy for gerrit to 10 days https://review.openstack.org/234375 | 18:08 |
| yolanda | fungi, change sent | 18:08 |
| *** jsavak has quit IRC | 18:08 | |
| clarkb | hrm annegentle left http://developer.openstack.org/api-ref-objectstorage-v1.html is missing subdir info /me goes to file a bug | 18:08 |
| *** maishsk has quit IRC | 18:09 | |
| *** jsavak has joined #openstack-infra | 18:09 | |
| fungi | yolanda: oh, wait, i think i misunderstood what you were saying about the config for it | 18:09 |
| fungi | yolanda: i thought you were saying that it was configured to only rotate every 30 days but it looks like it's configured to rotate every day and retain 30 days of history | 18:09 |
| yolanda | yes | 18:10 |
| *** cdoron has quit IRC | 18:10 | |
| fungi | so yes, i believe the bug i'm correcting did prevent it from rotating | 18:10 |
| fungi | it had several weeks of logs in one 7gb file | 18:10 |
| fungi | so i think it was trying and failing to rotate every day since that change merged | 18:10 |
| *** Sukhdev has joined #openstack-infra | 18:11 | |
| *** EricGonczer_ has joined #openstack-infra | 18:11 | |
| fungi | the current 30 day retention of daily rotated files should be fine | 18:11 |
| fungi | i don't think we need 234375 | 18:11 |
| fungi | sorry about that | 18:11 |
| yolanda | ah no problem, it was a simple change | 18:11 |
| *** maishsk has joined #openstack-infra | 18:12 | |
| yolanda | fungi, so now i'm happy that i can babysit these things.. as it's much easier to land a change and watch carefully the effects, and correct before it has any side effect | 18:12 |
| fungi | for the pypi mirrors, we likely need some other shorter rotation schedule than daily | 18:12 |
| *** shardy_ has joined #openstack-infra | 18:12 | |
| fungi | or lots of extra room for apache logs | 18:13 |
| *** shardy has quit IRC | 18:13 | |
| *** woodster_ has joined #openstack-infra | 18:13 | |
| *** dprince has joined #openstack-infra | 18:13 | |
| clarkb | filed https://bugs.launchpad.net/openstack-api-site/+bug/1505768 | 18:14 |
| openstack | Launchpad bug 1505768 in openstack-api-site "Swift subdir response info not documented" [Undecided,New] | 18:14 |
| *** e0ne has joined #openstack-infra | 18:17 | |
| *** mriedem has quit IRC | 18:18 | |
| *** shardy_ has quit IRC | 18:18 | |
| *** sabeen1 has joined #openstack-infra | 18:18 | |
| odyssey4me | fungi there are pypi mirrors in -infra? how do I use them? | 18:18 |
| *** shardy has joined #openstack-infra | 18:18 | |
| fungi | odyssey4me: they're provider and region specific, for use in our jobs | 18:18 |
| *** ociuhandu has quit IRC | 18:19 | |
| fungi | odyssey4me: our job workers are configured to use the closest mirror by default | 18:19 |
| odyssey4me | fungi even better - is there a nodepool file with the details in it in the image? | 18:19 |
| odyssey4me | fungi I want to read it and transfer that config into the containers | 18:19 |
| fungi | odyssey4me: the pip configuration in the vm, yes | 18:19 |
| clarkb | odyssey4me: /etc/pip/pip.conf | 18:19 |
| clarkb | iirc | 18:19 |
| clarkb | or are we still per user because distutils? | 18:20 |
| odyssey4me | fungi clarkb ah, makes sense - so instead of blowing that away I'll back it up first and consume the details in it :) | 18:20 |
| odyssey4me | thanks! | 18:20 |
| *** fawadkhaliq has quit IRC | 18:21 | |
| *** mriedem has joined #openstack-infra | 18:21 | |
| *** fawadkhaliq has joined #openstack-infra | 18:21 | |
| *** geoffarn_ has joined #openstack-infra | 18:22 | |
| *** claudiub has joined #openstack-infra | 18:22 | |
| *** sabeen1 has quit IRC | 18:22 | |
| mordred | clarkb, odyssey4me: I believe we copy pip.conf into each user's home dir | 18:23 |
| mordred | so it goes into root, jenkins and stack, last I checked | 18:24 |
| *** fawadkhaliq has quit IRC | 18:24 | |
| mordred | (same with .pydistutils.conf or whatever it is) | 18:24 |
| *** fawadkhaliq has joined #openstack-infra | 18:24 | |
| *** fawadkhaliq has quit IRC | 18:24 | |
| *** achanda has quit IRC | 18:25 | |
| *** geoffarnold has quit IRC | 18:26 | |
| *** achanda has joined #openstack-infra | 18:27 | |
| *** sergmelikyan has quit IRC | 18:27 | |
| *** maishsk has quit IRC | 18:29 | |
| *** e0ne has quit IRC | 18:30 | |
| *** dizquierdo has quit IRC | 18:30 | |
| *** maishsk has joined #openstack-infra | 18:30 | |
| *** sergmelikyan has joined #openstack-infra | 18:30 | |
| *** maishsk has quit IRC | 18:30 | |
| *** Sukhdev has quit IRC | 18:31 | |
| *** sergmelikyan has quit IRC | 18:31 | |
| *** skylerberg has quit IRC | 18:31 | |
| *** esmute__ has quit IRC | 18:32 | |
| *** kdas_ has joined #openstack-infra | 18:32 | |
| *** Sukhdev has joined #openstack-infra | 18:33 | |
| notmyname | clarkb: the idea of doing extra requests do different objects on a listing request seems pretty inefficient. so I'm not in love with that idea | 18:33 |
| notmyname | clarkb: however, the container is storing the last modified time for every object in it | 18:33 |
| notmyname | clarkb: so there might be a way to ask the container for the max() of all of the objects that have a given prefix | 18:34 |
| clarkb | notmyname: does the current prefix listing have to iterate all those object anyways? | 18:34 |
| clarkb | notmyname: or will this incur some additional overhead to the prefix listing? | 18:34 |
| *** kdas_ has quit IRC | 18:34 | |
| notmyname | no, actually it doesn't. it's fairly clever about how it's getting pages of results back with prefix/delimiter listings, so it skips over stuff that won't be returned | 18:35 |
| greghaynes | notmyname: inefficient in which way? it should be super fast on reads and its not bad on writes... so just number of wasted objects? | 18:35 |
| notmyname | clarkb: ie adding this would add additional overhead | 18:35 |
| *** sergmelikyan has joined #openstack-infra | 18:35 | |
| AJaeger | anteaya, fungi, jeblair : Everybody loves RTD ;( - see https://review.openstack.org/234336 ... Do you have some suggestions for better text for the commit message? | 18:35 |
| *** salv-orlando has quit IRC | 18:35 | |
| *** kushal has quit IRC | 18:35 | |
| notmyname | greghaynes: requesting another object is at least one and maybe more network requests with corresponding disk hits | 18:36 |
| anteaya | AJaeger: I don't think massaging the commit message with net you any additional happy customers on this patch | 18:36 |
| *** amuller is now known as amuller_afk | 18:36 | |
| notmyname | clarkb: however, I'd like to step back to the actual use case | 18:36 |
| notmyname | clarkb: one way to phrase it is "make it look like apache/nginx". but what are you doing with that? just trying to find most recent data right? | 18:37 |
| notmyname | so the use case is alternatively phrased as "list by last modified", right? | 18:37 |
| clarkb | notmyname: yes, to add in the navigation of build logs stored in swift | 18:37 |
| clarkb | s/add/aid/ | 18:37 |
| clarkb | notmyname: a specific example would be to differentiate by timestamp many periodic (hourly/daily/weekly etc) test run logs | 18:38 |
| greghaynes | notmyname: Yea, I think if you did it you wouldnt want to make all reads by default request that data... | 18:38 |
| * greghaynes is also all ears for other options | 18:38 | |
| anteaya | AJaeger: I position still stands by the way | 18:38 |
| anteaya | AJaeger: s/I/my | 18:38 |
| *** notnownikki has joined #openstack-infra | 18:39 | |
| *** ashleighfarnham has joined #openstack-infra | 18:39 | |
| AJaeger | anteaya: yep. My point is that they think that this will disable rtd publishing but it does not. They can still pull from there... | 18:39 |
| greghaynes | AJaeger: Ah, good point | 18:39 |
| notmyname | clarkb: in order to find the last log written? or what's the reason you want to differentiate the logs by timestamp? | 18:40 |
| anteaya | AJaeger: ah okay yes then you are correct, perhaps a clearer commit message is in order | 18:41 |
| clarkb | notmyname: so that while drinking morning cup of coffee you can find the most recent job run for your pet test rig | 18:41 |
| anteaya | AJaeger: you have a way with clear git commit messages, I clear space for your compositional ministrations | 18:41 |
| AJaeger | anteaya: let me try... | 18:41 |
| * anteaya lets AJaeger try | 18:41 | |
| fungi | notmyname: or to find the periodic job log from four days ago, or find one from the previous month to compare against | 18:41 |
| clarkb | notmyname: http://logs.openstack.org/periodic/swift-propose-translation-update/ would be an example | 18:41 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Remove cleanup cruft from security.openstack.org https://review.openstack.org/234383 | 18:42 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Standardize vhost options for security.o.o https://review.openstack.org/234384 | 18:42 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Rename security.o.o vhost template for reusability https://review.openstack.org/234385 | 18:42 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Add HTTPS support to tarballs.openstack.org https://review.openstack.org/234386 | 18:42 |
| clarkb | those run daily, you would sort by last modified to find the logs that you want | 18:42 |
| clarkb | er looks like weekl | 18:42 |
| clarkb | ish | 18:42 |
| notmyname | clarkb: fungi: got it. thanks | 18:42 |
| *** jsavak has quit IRC | 18:42 | |
| *** geoffarnold has joined #openstack-infra | 18:43 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Do not add rtfd-hooks for murano, rally, sahara https://review.openstack.org/234336 | 18:43 |
| *** geoffarn_ has quit IRC | 18:43 | |
| AJaeger | greghaynes, anteaya: Is that commit message clearer? | 18:43 |
| notmyname | so instead of "make it have the exact same UI as nginx", the problem to solve is to sort the results by last modified time | 18:43 |
| greghaynes | AJaeger: Yea, I really wasnt aware of the rtd pulling, but that really changes the intent here :) | 18:44 |
| *** jsavak has joined #openstack-infra | 18:44 | |
| *** armax has joined #openstack-infra | 18:45 | |
| AJaeger | greghaynes: I still think that there's no benefit in having it on two places but can't stop you ;) | 18:45 |
| greghaynes | SEO is really important IMO for docs | 18:45 |
| greghaynes | without it users are lost | 18:46 |
| greghaynes | but, not an infra discussion I think | 18:46 |
| clarkb | notmyname: yes, based on feedback from our users | 18:46 |
| anteaya | AJaeger: it is for me | 18:46 |
| notmyname | clarkb: ok. that makes sense | 18:46 |
| jeblair | greghaynes: rtd updates once a day | 18:46 |
| clarkb | notmyname: but I think we also need the timestamp as well (its not just a sort) so that users can see when it ran too | 18:47 |
| clarkb | notmyname: sort for scanning easily, and specific timestamp for concrete dat | 18:47 |
| AJaeger | greghaynes: so, how can we improve SEO for docs.o.o? I think that's the question we should ask... | 18:47 |
| jeblair | greghaynes: this change does not stop rtd from doing whatever it is they do. | 18:48 |
| * anteaya notes 13 minutes to whopner, goes to refresh her tea | 18:48 | |
| *** notnownikki has quit IRC | 18:48 | |
| openstackgerrit | Mikhail S Medvedev proposed openstack-infra/puppet-openstackci: Add pass through for project-config revision https://review.openstack.org/198546 | 18:48 |
| greghaynes | jeblair: Yea, I kind of missed that. I am totally on board with telling people to use $other_option for that | 18:48 |
| jeblair | greghaynes: what $other_option? | 18:48 |
| greghaynes | jeblair: have rtd pull daily | 18:49 |
| jeblair | greghaynes: is not an option | 18:49 |
| jeblair | greghaynes: it _is what rtd does_ | 18:49 |
| greghaynes | oh, we just host and have them pull from us. hah | 18:49 |
| fungi | yeah, it works by defauly | 18:49 |
| fungi | default too | 18:49 |
| *** ybathia has quit IRC | 18:50 | |
| greghaynes | AJaeger: Yes, I think that is a good thing to work on (improving docs.o.o). I am not sure spending a lot of effort to improve SEO is worth it if there is a solution out there already though | 18:51 |
| *** ivar-lazzaro has joined #openstack-infra | 18:51 | |
| greghaynes | AFAICT the big risk is confusion which can be mitigated by pointing people at official docs in the docs themselves if we want | 18:51 |
| jeblair | greghaynes: yep, i think that's what we should do | 18:52 |
| *** abregman has joined #openstack-infra | 18:52 | |
| *** erikwilson has joined #openstack-infra | 18:52 | |
| *** ZZelle_ has joined #openstack-infra | 18:53 | |
| *** sergmelikyan has quit IRC | 18:55 | |
| *** jsavak has quit IRC | 18:56 | |
| jhesketh | Morning. I have to tender my apologies for the meeting | 18:57 |
| jhesketh | (not feeling very well) | 18:57 |
| jhesketh | Will catch up on the minutes later | 18:57 |
| clarkb | jhesketh: get better | 18:57 |
| jhesketh | Thanks | 18:58 |
| *** dramalho has quit IRC | 18:58 | |
| *** jsavak has joined #openstack-infra | 18:58 | |
| reed | mordred, do you think there is any way to use openstack client *without* keystone? | 18:59 |
| anteaya | jhesketh: get some sleep | 18:59 |
| fungi | meeting time! | 19:00 |
| *** tonytan4ever has joined #openstack-infra | 19:00 | |
| stevemar_ | reed: you mean not a having to authenticate? | 19:00 |
| stevemar_ | reed: it mostly depends on a working keystone, yes | 19:01 |
| stevemar_ | dtroyer: ^ | 19:01 |
| *** glauco_ has joined #openstack-infra | 19:01 | |
| *** glauco___ has quit IRC | 19:01 | |
| *** DericHorn-HP has joined #openstack-infra | 19:01 | |
| stevemar_ | reed: you can create some commands that bypass authentication | 19:01 |
| mordred | stevemar_, reed: there is a no-auth mode | 19:02 |
| *** amotoki has quit IRC | 19:02 | |
| mordred | stevemar_, reed: if you set auth-type to None - it will use the keystoneauth admin-token plugin with bogus values and will completely skip keystone | 19:02 |
| mordred | you obviously need to configue the endpoint url at that point | 19:02 |
| *** dzamboni has quit IRC | 19:03 | |
| stevemar_ | mordred: o right, forgot about that, we did that for ironic no? | 19:03 |
| mordred | yup | 19:03 |
| stevemar_ | bueno | 19:03 |
| *** sergmelikyan has joined #openstack-infra | 19:03 | |
| *** sergmelikyan has quit IRC | 19:03 | |
| *** bcourt has quit IRC | 19:03 | |
| mordred | I would imagine it would also make it good for openstackless-swift, but I don't know of anyone using shade with swift in a non-openstack context yet | 19:03 |
| reed | stevemar_, mordred: I'm confused... how will it know where to connect then? | 19:04 |
| * reed goes and read the docs | 19:04 | |
| *** geoffarnold has quit IRC | 19:04 | |
| anteaya | AJaeger: jeblair fungi any objection to me approving https://review.openstack.org/#/c/234336/ ? | 19:04 |
| *** geoffarnold has joined #openstack-infra | 19:04 | |
| dtroyer | using existing commands without keystone requires both —os-url and —os-token to be set. that also skips the service catalog lookup | 19:05 |
| *** sambetts has quit IRC | 19:05 | |
| reed | openstack: error: argument --os-auth-type: invalid choice: 'None' | 19:05 |
| *** carl_baldwin has joined #openstack-infra | 19:05 | |
| dtroyer | the admin-token auth type is similar but not quite (yet) the same thing as OSC has done from the beginning | 19:05 |
| dtroyer | I don't think we allow none there | 19:06 |
| reed | indeed, you don't :) | 19:06 |
| *** rkukura has joined #openstack-infra | 19:06 | |
| mordred | oh right. osc | 19:07 |
| mordred | sorry - I thought you were talking about shade | 19:07 |
| mordred | "None" is a shade hack | 19:07 |
| reed | mordred, before heading to Shade, I wanted to test osc | 19:07 |
| *** nadya has joined #openstack-infra | 19:07 | |
| reed | felt more like 'baby step' | 19:07 |
| dtroyer | if you need to set an auth type, token_endpoint will force the behaviour I mentioned above. That's what you get by supplying both —os-uel and --os-token | 19:07 |
| dtroyer | OSC does things that shade doesn't, things like this | 19:08 |
| *** sambetts has joined #openstack-infra | 19:08 | |
| reed | dtroyer, I managed to move forwars, no I get 403 error | 19:09 |
| *** abregman is now known as abregman|home | 19:09 | |
| * reed dives deeper to understand which credentials are required | 19:09 | |
| dtroyer | The URL you provide must be the actual endpoint URL you want, ie, like what the service catalog would have provided | 19:10 |
| *** notnownikki has joined #openstack-infra | 19:10 | |
| *** nadya_ has joined #openstack-infra | 19:11 | |
| *** nadya has quit IRC | 19:12 | |
| openstackgerrit | Merged openstack-infra/infra-specs: StoryBoard: Worklists and Boards https://review.openstack.org/202989 | 19:12 |
| *** sergmelikyan has joined #openstack-infra | 19:12 | |
| armax | not sure if this is right channel, but I see old content on the developer guide | 19:15 |
| reed | openstack --os-url https://objects.dreamhost.com/auth --os-username stemaf4:smaffulli --os-token $TOKEN container list < gives me 403 | 19:15 |
| armax | for instance: http://docs.openstack.org/developer/neutron/ shows an old snapshot what what we currently have in tree | 19:15 |
| armax | I wonder what might have happened | 19:16 |
| AJaeger | armax: how old? | 19:16 |
| armax | AJaeger: like at least a week or so | 19:16 |
| AJaeger | updated "Tue 13 Oct 2015 07:36:25 PM CEST" | 19:16 |
| armax | AJaeger: I did see the guide update based on the latest changes we made in the devref in tree | 19:16 |
| armax | AJaeger: I am baffled then | 19:16 |
| AJaeger | armax: so, why do you say it's old? Let's double check it... | 19:17 |
| armax | AJaeger: for instance I did this: https://github.com/openstack/neutron/commit/a853ff30183a594e8737ea395315451a58dc5b4e | 19:17 |
| armax | AJaeger: and I remember seeing in docs.o.i | 19:17 |
| armax | docs.o.o | 19:17 |
| armax | but now the content it’s gone | 19:17 |
| *** matrohon has joined #openstack-infra | 19:17 | |
| openstackgerrit | Merged openstack-infra/system-config: Revert "Revert "Set ovh max-servers to 0"" https://review.openstack.org/234341 | 19:18 |
| *** bcourt has joined #openstack-infra | 19:18 | |
| *** e0ne has joined #openstack-infra | 19:19 | |
| AJaeger | armax: that looks strange indeed | 19:20 |
| armax | AJaeger: this is the latest that I recall: http://docs-draft.openstack.org/19/231819/2/check/gate-neutron-docs/ee68c34//doc/build/html/policies/index.html | 19:20 |
| * AJaeger builds neutron docs locally... | 19:20 | |
| armax | AJaeger: and I did see it published | 19:20 |
| armax | AJaeger: I am pretty positive | 19:20 |
| AJaeger | I believe you... | 19:20 |
| * armax too | 19:20 | |
| AJaeger | armax footer has " Last updated on Sun Oct 11 17:47:34 2015, commit b175e03. Created using Sphinx 1.2.3. " | 19:21 |
| AJaeger | and that's a kilo commit, isn't it? | 19:21 |
| *** glauco___ has joined #openstack-infra | 19:21 | |
| armax | checking | 19:21 |
| *** ivar-laz_ has joined #openstack-infra | 19:22 | |
| armax | AJaeger: hang on, tox -edocs breaks for me locally | 19:22 |
| AJaeger | armax: for me as well... | 19:22 |
| armax | AJaeger: I wonder why that wasn’t caught | 19:22 |
| armax | by the docs job | 19:22 |
| AJaeger | armax: so, that's one part of the story... | 19:22 |
| AJaeger | armax: we don't use tox target, we invoke build_sphinx directly, without the -W | 19:23 |
| armax | I see | 19:23 |
| armax | AJaeger: might that be reason for the stale content? | 19:23 |
| armax | AJaeger: it shouldn’t be, right? | 19:23 |
| armax | AJaeger: btw I can’t seem to find that commit | 19:24 |
| *** glauco_ has quit IRC | 19:24 | |
| *** mwagner_ has quit IRC | 19:24 | |
| *** ivar-lazzaro has quit IRC | 19:25 | |
| AJaeger | armax: still, it shouldn't - checking further... | 19:25 |
| armax | AJaeger: agreed | 19:25 |
| armax | AJaeger: fixing btw | 19:25 |
| bltavares | hey there folks, sorry to bother again. we have a few acceptance tests for provisioning waiting another +2/+A. Would someone be able to help us with that? | 19:26 |
| *** geoffarn_ has joined #openstack-infra | 19:26 | |
| clarkb | the docs job does not use tox -edocs | 19:26 |
| armax | clarkb: that’s what I figured | 19:26 |
| AJaeger | armax: I thought warn_errors would enable this but doesn't - better check for that separately | 19:27 |
| armax | AJaeger: so you suggest to keep it as is? | 19:27 |
| SpamapS | pabelanger: http://docs.openstack.org/infra/system-config/infra-cloud.html <-- start here. :) | 19:27 |
| pabelanger | SpamapS: ack | 19:28 |
| AJaeger | armax: http://docs.openstack.org/developer/neutron/kilo/ exists as well.. | 19:28 |
| mordred | reed: what are you trying to accomplish? | 19:28 |
| AJaeger | armax: I mean: Check why python setup.py build_sphinx does not error out... | 19:28 |
| reed | mordred, simply authenticate to dreamhost ceph at this point :) | 19:28 |
| armax | AJaeger: I am on it | 19:29 |
| reed | mordred, using the openstack client, not swift client (that one works) | 19:29 |
| mordred | reed: gotcha. and it doesn't use keystone for auth? | 19:29 |
| SpamapS | pabelanger: and we've been doing an o-k job recording what we're up to here https://etherpad.openstack.org/p/infra-cloud-bootstrap | 19:29 |
| reed | mordred, nope, unfortunately | 19:29 |
| mordred | reed: k. interesting | 19:29 |
| *** geoffarnold has quit IRC | 19:29 | |
| openstackgerrit | Merged openstack-infra/shade: Make attach_instance return updated volume object https://review.openstack.org/232830 | 19:30 |
| SpamapS | crinkle: ^ lets make sure our vlan woes are documented somehwere (patches to system-config docs seem like a good way to get it out of your head before we get it 100% automated.. unless you already automated it .. and then never mind ;) | 19:30 |
| reed | and as with all openstack things, it's complicated :) | 19:30 |
| crinkle | SpamapS: nope not automated, need a couple of things in bifrost and glean plus the init script thing that we need | 19:31 |
| AJaeger | armax: http://docs-draft.openstack.org/19/231819/2/check/gate-neutron-docs/ee68c34//doc/build/html/policies/blueprints.html has neutron 8... as version while the docs.openstack.org has 2015.1.2, so something is indeed broken. | 19:32 |
| *** kushal has joined #openstack-infra | 19:32 | |
| pabelanger | SpamapS: I might be able to access some bare metal locally, otherwise can't do much that part. I'll review eitherpad and see what I can do to help | 19:32 |
| armax | AJaeger: um I see | 19:33 |
| *** sergmelikyan has quit IRC | 19:33 | |
| AJaeger | armax: we might need some help from others - but right now is infra-meeting... | 19:33 |
| armax | AJaeger: so this must have happened after the release | 19:33 |
| SpamapS | pabelanger: we can get you onto these boxes. | 19:33 |
| armax | AJaeger: ok no worries…this can wait | 19:33 |
| SpamapS | pabelanger: *they are for us* | 19:33 |
| *** sergmelikyan has joined #openstack-infra | 19:34 | |
| *** notnownikki has quit IRC | 19:34 | |
| armax | AJaeger: btw python setup.py build_sphinx does fail locally too | 19:34 |
| reed | success! | 19:35 |
| pabelanger | SpamapS: okay, let me read tonight and ping you tomorrow about next steps | 19:35 |
| reed | mordred, playing around with auth-url I found the right one :) | 19:35 |
| mordred | reed: woot! | 19:35 |
| reed | openstack --os-url https://objects.dreamhost.com/ --os-username 'stemaf4:smaffulli' --os-token $TOKEN container list | 19:36 |
| SpamapS | pabelanger: the biggest thing we need help with is iterating on the weird vlan setup we have in those racks. | 19:36 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Remove extra install of release pkgs in fedora-minimal https://review.openstack.org/227131 | 19:36 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Reset yum/dnf cache to correct location https://review.openstack.org/232863 | 19:36 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Add Fedora 22 support to yum-minimal https://review.openstack.org/227127 | 19:36 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Update default fedora-minimal to f22 https://review.openstack.org/227126 | 19:36 |
| stevemar_ | reed: hmmm you shouldn't be able to use token+url AND username | 19:36 |
| mtreinish | clarkb, jeblair, fungi: when you get a sec can you take a look at: https://review.openstack.org/232187 | 19:36 |
| pabelanger | SpamapS: sure. I'll start there | 19:36 |
| SpamapS | pabelanger: once that is done, we will also need to start testing nodepool pointed at this cloud. | 19:36 |
| reed | stevemar_, ahahah probably one is ignored? | 19:37 |
| mtreinish | that's needed to start pruning the subunit2sql db | 19:37 |
| mtreinish | I imagine we're quite close to running out of space | 19:37 |
| mtreinish | since we were <10% free a couple weeks ago | 19:37 |
| SpamapS | pabelanger: sync with crinkle on that. They hand-built a solution, but now we need to make it happen automatically when we boot boxes. :) | 19:37 |
| *** gyee has quit IRC | 19:37 | |
| *** xyang1 has joined #openstack-infra | 19:38 | |
| pabelanger | ack | 19:38 |
| greghaynes | I think crinkle was trying to do the static baked in eni config for now so we can get terrible-cloud up and rolling | 19:38 |
| pabelanger | SpamapS: I'll start allocating time to get more involved | 19:38 |
| *** baoli has quit IRC | 19:39 | |
| *** marzif has joined #openstack-infra | 19:39 | |
| *** nadya_ has quit IRC | 19:41 | |
| *** devananda has joined #openstack-infra | 19:41 | |
| SpamapS | pabelanger: great thanks. | 19:42 |
| SpamapS | pabelanger: whats your TZ? | 19:42 |
| SpamapS | we were going to try and sync up last week and got distracted | 19:43 |
| SpamapS | might make sense to do that some time today | 19:43 |
| *** e0ne has quit IRC | 19:43 | |
| SpamapS | as I'd like to make sure we keep pushing forward with the whole thing | 19:43 |
| pabelanger | SpamapS: EDT | 19:44 |
| SpamapS | crinkle: you available for a sync-up in 45 minutes? | 19:45 |
| SpamapS | greghaynes: ^ ? | 19:45 |
| crinkle | SpamapS: yep | 19:45 |
| SpamapS | pabelanger: ^? | 19:45 |
| SpamapS | I need to get some lunch in me | 19:45 |
| zul | can someone allow my email to openstack-announce go through please? | 19:45 |
| greghaynes | uhhh, I also need food | 19:45 |
| *** mestery has joined #openstack-infra | 19:45 | |
| SpamapS | PDX wizard, needs food, badly. | 19:45 |
| SpamapS | LA wizard shot the food. | 19:46 |
| *** cloudtrainme has joined #openstack-infra | 19:46 | |
| greghaynes | 45min is 13:30 PST? I could probably have food by then | 19:46 |
| greghaynes | I think I am mostly up to speed with crinkle's status because of random chatting, but I'd still be up for joining | 19:47 |
| *** geoffarn_ has quit IRC | 19:47 | |
| *** Hal has quit IRC | 19:47 | |
| SpamapS | well maybe a better question to ask is, how can we help? | 19:47 |
| SpamapS | crinkle: ^ ? | 19:47 |
| SpamapS | Bifrost and glean stuff needs to land? | 19:48 |
| crinkle | i don't think glean stuff is ready, but i stopped playing with it in the interest of "get cloud up" | 19:48 |
| mordred | which glean stuff? | 19:48 |
| crinkle | bifrost stuff needs to land | 19:48 |
| pabelanger | SpamapS: tuesdays are a bad day for me. Book with meetings the rest of the day | 19:48 |
| mordred | oh - the vlan stuff? | 19:48 |
| greghaynes | yea | 19:48 |
| pabelanger | SpamapS: any other day works | 19:48 |
| crinkle | mordred: yeah | 19:48 |
| SpamapS | pabelanger: no worries, I think we'll have more for you tomorrow anyway :) | 19:49 |
| SpamapS | OH but were we going to table that and just bake in vlan25? | 19:49 |
| greghaynes | SpamapS: yep | 19:49 |
| *** hodos has joined #openstack-infra | 19:49 | |
| crinkle | right - BUT | 19:49 |
| * greghaynes waits in anticipation | 19:49 | |
| crinkle | i have been having hardware issues and haven't deployed new iamges in a few days | 19:49 |
| greghaynes | ah | 19:49 |
| SpamapS | noooooooooooooooooooooo | 19:49 |
| crinkle | so, just using waht greg had already deployed a while ago | 19:50 |
| crinkle | documented what i had: https://etherpad.openstack.org/p/infra-cloud-inventory-status | 19:50 |
| SpamapS | crinkle: ok, so is there anything you think we can do to help with _that_ ? | 19:50 |
| greghaynes | crinkle: the devuser cant sudo is just an old image I think? | 19:50 |
| anteaya | zul: the admin on that list is ttx | 19:50 |
| greghaynes | crinkle: that looks about right for ratio of broken to not, actually | 19:51 |
| crinkle | greghaynes: yeah, but if i can't redeploy then it affects ability to cloud | 19:51 |
| SpamapS | or are you just at the bottom of the pit of sadness and we should mostly add encouraging words and stay off the rim so less stuff comes tumbling downward? | 19:51 |
| greghaynes | crinkle: hrm | 19:51 |
| crinkle | SpamapS: help with hardware stuff would be great! at the bottom of the list are the ones i found issues with | 19:51 |
| ttx | zul: already approved | 19:51 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Do not add rtfd-hooks for murano, rally, sahara https://review.openstack.org/234336 | 19:51 |
| SpamapS | crinkle: Unfortunately, I can't open HP tickets anymore... ;) | 19:51 |
| AJaeger | anteaya: the rtfd change didn't pass testing, so had to update it... | 19:52 |
| SpamapS | crinkle: but I _can_ advise you if you haven't done that. | 19:52 |
| greghaynes | crinkle: I have to take care of a couple things but later this afternoon I could probably poke at the hardware a bit | 19:52 |
| crinkle | SpamapS: of course, but debugging help would be great | 19:52 |
| anteaya | AJaeger: very good | 19:52 |
| crinkle | i'm just starting to check other machines to see if they have the same issue | 19:52 |
| crinkle | but good news is there is a slightly bigger cloud now | 19:52 |
| SpamapS | crinkle: sure. VSP stuck and won't power on, btw, should definitely just be tickets. | 19:52 |
| greghaynes | crinkle: What I would do is make an image and try to deploy to everything but the bastion and current bm host, see what sticks, and then make a known new list of ok hardware, unenroll everything but stuff on that lsit | 19:53 |
| SpamapS | crinkle: ask for update of ilo firmware, that usually does it. Sometimes they also end up replacing motherboards ... :P | 19:53 |
| greghaynes | and then just charge ahead | 19:53 |
| sweston | asselin: https://review.openstack.org/#/c/234404/ | 19:53 |
| devananda | ohhai. did something change with the doc jobs and I only now noticed? http://docs.openstack.org/developer/ironic/ is showing the kilo/stable output, not master | 19:53 |
| sweston | asselin: changes to the database for gerrit change queries | 19:53 |
| sweston | asselin: unfortunately, this is all I have time for this week | 19:54 |
| crinkle | greghaynes: ya...just got spooked since a machine that had been working fine previously seemed to die, and the next two i tried had the same issue | 19:54 |
| SpamapS | crinkle: anyway, it sounds like we just need to boot the image that we think will work with the hardware we've got. | 19:54 |
| greghaynes | crinkle: ah | 19:54 |
| SpamapS | crinkle: ugh that would frustrate me too | 19:54 |
| *** marzif has quit IRC | 19:54 | |
| crinkle | so...probably gonna do one at a time rather than blow everything away | 19:54 |
| asselin__ | sweston, cool, did you deploy those changes to your server? | 19:54 |
| *** nadya has joined #openstack-infra | 19:54 | |
| *** coreycb` is now known as coreycb | 19:55 | |
| sweston | asselin__: no, they're on a development server right now, but I will put it in production later in the week | 19:55 |
| asselin__ | sweston, ok, ping me when you do | 19:56 |
| greghaynes | crinkle: hrm, if you want I could poke at the box you think is a new failure? | 19:56 |
| sweston | asselin__: will do, probably on thursday or friday | 19:56 |
| crinkle | greghaynes: sure | 19:56 |
| greghaynes | crinkle: or, did you debug what the failure is? | 19:56 |
| crinkle | greghaynes: nope, just won't power on | 19:56 |
| greghaynes | sweet | 19:56 |
| SpamapS | crinkle: does that mean that you have an image with baked-in vlan25 already? And we just need the init script that moves the IP from there -> veths? | 19:57 |
| *** ivar-laz_ has quit IRC | 19:57 | |
| *** ivar-lazzaro has joined #openstack-infra | 19:57 | |
| greghaynes | SpamapS: We can bake in the veths too | 19:57 |
| greghaynes | I think | 19:57 |
| *** jsavak has quit IRC | 19:58 | |
| greghaynes | SpamapS: Can you bake in veth in eni? | 19:58 |
| *** Sukhdev has quit IRC | 19:58 | |
| SpamapS | greghaynes: yeah | 19:58 |
| *** jsavak has joined #openstack-infra | 19:58 | |
| greghaynes | Then yea, no scripts needed | 19:58 |
| SpamapS | greghaynes: but glean is going to put the IP on vlan25 | 19:58 |
| *** dramalho has joined #openstack-infra | 19:58 | |
| SpamapS | and the bridge won't be setup | 19:58 |
| *** annegentle has joined #openstack-infra | 19:58 | |
| *** annegent_ has joined #openstack-infra | 19:58 | |
| SpamapS | greghaynes: oh hm, I see what you're getting at | 19:58 |
| *** annegentle has quit IRC | 19:58 | |
| greghaynes | yea, were violating layers | 19:58 |
| SpamapS | so yeah we could bake the whole bridge setup in too | 19:58 |
| SpamapS | no we can do it | 19:59 |
| SpamapS | it's totally a violation | 19:59 |
| *** annegent_ has quit IRC | 19:59 | |
| greghaynes | yep | 19:59 |
| SpamapS | but it will work | 19:59 |
| greghaynes | and were gonna do it | 19:59 |
| greghaynes | :) | 19:59 |
| *** ivar-lazzaro has quit IRC | 19:59 | |
| SpamapS | awww yeah | 19:59 |
| *** annegentle has joined #openstack-infra | 19:59 | |
| SpamapS | just like $240 worth o pudding | 19:59 |
| *** dramalho has quit IRC | 19:59 | |
| *** dramalho has joined #openstack-infra | 19:59 | |
| *** ivar-lazzaro has joined #openstack-infra | 19:59 | |
| *** alivigni has quit IRC | 20:00 | |
| greghaynes | crinkle: got a host uuid? | 20:00 |
| crinkle | greghaynes: try 7889d9ff-5c17-4f2f-9ac8-8f7b9e8f3bac | 20:00 |
| SpamapS | ok, so we need to write a static ENI with the veths and bridges configured... | 20:00 |
| SpamapS | and then change bifrost's config to use the veth as the interface name | 20:01 |
| greghaynes | SpamapS: Yep. I actually did most everything needed, we just need to resurrect some of the work | 20:01 |
| *** nadya has quit IRC | 20:01 | |
| *** EricGonczer_ has quit IRC | 20:01 | |
| *** EricGonc_ has joined #openstack-infra | 20:01 | |
| greghaynes | SpamapS: I just made an element with that eni config and added it to the build | 20:01 |
| clarkb | yolanda: for the nodepool thing, couldn't you run nova-docker/magnum/ironic/etc instead of baking all of that stuff into nodepool? then nodepool could just speak shade to them to consume the resources? | 20:01 |
| *** geoffarnold has joined #openstack-infra | 20:02 | |
| *** ociuhandu has joined #openstack-infra | 20:02 | |
| *** Rockyg has joined #openstack-infra | 20:02 | |
| SpamapS | greghaynes: sounds good. I'll just be over here eating food | 20:02 |
| *** sridhar_ram1 has joined #openstack-infra | 20:02 | |
| SpamapS | greghaynes: I'll be back in a bit. ;) | 20:02 |
| greghaynes | oh crap, that food thing | 20:02 |
| clarkb | yolanda: and for the cse of providers not providing it you either run it yourself if its valuable or use what resources are available instead. | 20:02 |
| *** sridhar_ram has quit IRC | 20:03 | |
| yolanda | clarkb, i'm not saying that nodepool should implement that internally, but that should be architected in a way that you can grow it | 20:04 |
| zul | ttx: thanks | 20:04 |
| yolanda | so in case of a close scenario, where you have to rely on a provider with certain limitations, you can extend with freedom depending on your needs | 20:04 |
| clarkb | yolanda: ok, I was under the impression that it would just work today as is assuming you used the nova apis | 20:04 |
| AJaeger | do we have a policy for IRC channel names - storlets is not part of the big tent and asks whether to use #storlets or #openstack-storlets, see https://review.openstack.org/#/c/234350/ | 20:05 |
| *** bcourt has quit IRC | 20:05 | |
| clarkb | yolanda: so in that case it would be nova + xen/kvm, nova + docker, nova + ironic | 20:05 |
| dougwig | AJaeger: #openstack- has some special ownership by infra, and can reclaim from lost admins and such. but i'm also curious to the answer, because all sorts get approved for the bots. | 20:05 |
| anteaya | AJaeger: jeblair has freenode acknowledgement for the #openstack- namespace | 20:06 |
| *** pelix has quit IRC | 20:06 | |
| clarkb | I briefly looked at what would be necessary to use it with rax onmetal and hte only problem there is onmetal doesn't allow custom images or snapshots at all | 20:06 |
| clarkb | but thats an implementation detail from whta Ican tell | 20:06 |
| anteaya | AJaeger: so because of that we encourage all channels that want to be using our tools to use the #openstack- prefix whether they are listed in the governance repo or not | 20:06 |
| yolanda | clarkb but what about the close scenarios, of a provider just supporting nova + kvm for example, and no plans to support more? | 20:07 |
| anteaya | AJaeger: the only exception is a channel that clearly has nothing to do with openstack and jsut wants to use our tools | 20:07 |
| AJaeger | anteaya: thanks for the info. Could you comment on 234350, please? | 20:07 |
| anteaya | AJaeger: can do | 20:07 |
| clarkb | yolanda: you would run your own nova+docker | 20:07 |
| clarkb | yolanda: and in that case you have no baremetal anyways so the ironic/baremetal case is a noop | 20:07 |
| *** geoffarnold has quit IRC | 20:08 | |
| clarkb | yolanda: I think this is important because tools like nova, nova-docker, ironic have already solved the problem space and trying to reinvent that wheel will lead to unnecessary effort in nodepool | 20:08 |
| clarkb | yolanda: I am positive tthat running a nova-docker is easier than teaching nodepool to manage containers itself | 20:08 |
| *** geoffarnold has joined #openstack-infra | 20:08 | |
| openstackgerrit | Clint Adams proposed openstack-infra/project-config: WIP - testing the rename scripts https://review.openstack.org/229601 | 20:08 |
| greghaynes | crinkle: oh, is that the one I was using before to iterate on new bm host? | 20:09 |
| greghaynes | crinkle: -24 seems like a familiar number... | 20:09 |
| *** glauco__ has joined #openstack-infra | 20:09 | |
| yolanda | clarkb but with your proposal you need to tweak settings on openstack provider, needing control and admin rights for that | 20:10 |
| anteaya | Clint: thanks, rather than listing them all in the commit message how would you feel about creating a text file with them and adding it to your scripts patch | 20:10 |
| crinkle | greghaynes: i have no idea | 20:10 |
| clarkb | yolanda: yes you would run your own | 20:10 |
| anteaya | Clint: and linking to the scripts patch in the commit message for the test run | 20:10 |
| Clint | anteaya: right | 20:10 |
| greghaynes | crinkle: I think it might be - the fact that it can TEXTCONS means we were using it for something | 20:10 |
| clarkb | yolanda: so in the docker cse you would go to your VM cloud and spin up capacity, run a control plane on a few nodes, point nodepool to it | 20:10 |
| Clint | anteaya: sure | 20:10 |
| anteaya | Clint: also never rebase your test run, always start new from master and just reuse the change id | 20:10 |
| anteaya | Clint: rebasing is a waste of time | 20:11 |
| crinkle | greghaynes: there's quite a few that can textcons | 20:11 |
| greghaynes | crinkle: there should be 4ish? | 20:11 |
| crinkle | greghaynes: way more than that | 20:11 |
| crinkle | greghaynes: i wrote them down https://etherpad.openstack.org/p/infra-cloud-inventory-status | 20:11 |
| *** amuller_afk is now known as amuller | 20:11 | |
| greghaynes | crinkle: huh. Can I nuke the inventory.json in the homedir or do you need that copied to something? | 20:11 |
| crinkle | greghaynes: feel free to nuke | 20:12 |
| *** bcourt has joined #openstack-infra | 20:12 | |
| openstackgerrit | Clint Adams proposed openstack-infra/project-config: WIP - testing the rename scripts https://review.openstack.org/229601 | 20:12 |
| Clint | anteaya: that's what i did | 20:12 |
| *** glauco___ has quit IRC | 20:12 | |
| anteaya | Clint: awesome | 20:12 |
| yolanda | clarkb and for that approach you can use any provider without admin rights, limitations? i'm always thinking in being flexible, and accomodate to downstream scenarios, that can be different from upstream | 20:12 |
| anteaya | Clint: much better thank you | 20:13 |
| anteaya | Clint: I will start going over that tomorrow | 20:13 |
| Clint | great | 20:13 |
| clarkb | yolanda: yes I don't see why you would be limited, if you can have nodepool talk to docker why couldn't you run a nova to do it instead? | 20:13 |
| clarkb | yolanda: basically if nodepool is able to do it then nova would be able to as well, same deal with baremetal | 20:13 |
| mgagne | zaro: did you take a look at the "new screen" in Gerrit? Looks like the old one is removed in 2.11.x I'm trying :-/ | 20:13 |
| fungi | mgagne: yeah, i think we're resigned to prying a few cold, dead fingers off the old change screen so that we're not stuck running gerrit 2.10 forever | 20:14 |
| Clint | obsolete, forked 2.10 | 20:15 |
| yolanda | clarkb i need to take a deeper look on it to discuss properly. I see that as a valid option as well, with the downside of having to maintain that deployment, but i can see as a possibility | 20:15 |
| *** mwagner_ has joined #openstack-infra | 20:15 | |
| clarkb | yolanda: well the upside is someone else is doing the hard work of writing the code | 20:15 |
| mgagne | fungi: right. so I'm curious about how openstack will be making it more "usable" | 20:15 |
| anteaya | mgagne: have you looked at gertty | 20:16 |
| clarkb | yolanda: so you have to run a service but you don't need to figure out all the variance in hardware to pxe/ipmi properly | 20:16 |
| mgagne | anteaya: the people I'm installing gerrit for is not the target audience for gertty | 20:16 |
| anteaya | mgagne: fair enough | 20:18 |
| Clint | how is the new screen less usable than the old screen? | 20:18 |
| AJaeger | Clint, anteaya, fungi: we have zuul/layout.yaml sorted: Split into sections, and each is sorted. Clint's change breaks that. | 20:18 |
| clarkb | mgagne: I think our plan is fix the bugs with it like broken unified diff, but we would accept the UI design decisions as currently made | 20:18 |
| anteaya | AJaeger: yes it does | 20:18 |
| fungi | mgagne: it's possible some interested developers might get the needed incentive to help upstream gerrit improve it | 20:18 |
| clarkb | Clint: it doesn't flow, you have to move up and down and left and right to review things | 20:18 |
| fungi | AJaeger: yep, we need to rethink how it's organized now that there's no namespace separation | 20:19 |
| anteaya | AJaeger: we turned of the testing for htat to get the change automated | 20:19 |
| AJaeger | how do we want to continue with the order in zuul/layout.yaml - and jenkins/jobs/projects.yaml? | 20:19 |
| mgagne | fungi: ok, it answers my question then =) | 20:19 |
| clarkb | Clint: with the old screen it was ]]]]] scroll down, comment | 20:19 |
| AJaeger | fungi, yep | 20:19 |
| AJaeger | anteaya: ah ;) | 20:19 |
| clarkb | Clint: current one is ]]]] scroll down, scroll up, attempt to comment, realize you need to reread something, scroll back down, repeat finally comment | 20:19 |
| Clint | hmm | 20:20 |
| anteaya | AJaeger: we couldn't figure out any other way, so yes post rename we can chat about how to organize it | 20:20 |
| AJaeger | I think we should have some order so that stuff can be found but it needs to be intuitive. Just alphabetical is enough... | 20:20 |
| AJaeger | anteaya: yes, my thinking as well... | 20:20 |
| fungi | AJaeger: i would be more in favor of just alpha order at this point, but we can do that as an easier follow-on change so as not to make the maintenance change less reviewable | 20:20 |
| anteaya | AJaeger: alphebetizing it is popular | 20:20 |
| clarkb | Clint: also its painful that the number one most requested feature is still missing (single page diff) | 20:20 |
| AJaeger | seems we're in agreement... | 20:20 |
| clarkb | Clint: but old lacked that too so hard to ocunt against new screen | 20:20 |
| anteaya | AJaeger: wonderful, thank you | 20:20 |
| *** abitha has quit IRC | 20:21 | |
| *** dramalho_ has joined #openstack-infra | 20:21 | |
| AJaeger | fungi: easy change - but one that will most probably break any change that is queued. Still, I agree to do it later as cleanup - and then push it through quickly... | 20:22 |
| *** glauco has joined #openstack-infra | 20:22 | |
| *** e0ne has joined #openstack-infra | 20:22 | |
| *** flepied has quit IRC | 20:24 | |
| greghaynes | crinkle: ok, resetting the ILO so see if that helps... | 20:24 |
| *** derekh has quit IRC | 20:24 | |
| * crinkle crosses fingers | 20:24 | |
| *** dramalho has quit IRC | 20:24 | |
| clarkb | Clint: oh also its super wide | 20:25 |
| *** glauco__ has quit IRC | 20:25 | |
| clarkb | I keep meaning to see what it looks like on my phone. old screen wasn't great but was useable | 20:25 |
| * Clint nods. | 20:25 | |
| *** baoli has joined #openstack-infra | 20:26 | |
| yolanda | clarkb, in terms of perfomance... how this will double nesting work? | 20:26 |
| clarkb | yolanda: it would be the same as what you would do otherwise right? | 20:26 |
| clarkb | yolanda: if you only have a VM cloud nad want to docker on it performance is the same if nodepool or nova does it | 20:27 |
| *** baoli has quit IRC | 20:27 | |
| greghaynes | crinkle: Yea, this looks like an actual hardware fail | 20:27 |
| greghaynes | crinkle: so \I/ | 20:27 |
| greghaynes | \O/ even | 20:27 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Do not add rtfd-hooks for murano, rally, sahara https://review.openstack.org/234336 | 20:27 |
| crinkle | greghaynes: :'( | 20:27 |
| yolanda | clarkb but if i follow you propose to run an openstack inside an openstack, then spin up docker instances on this second layer | 20:28 |
| *** shardy_ has joined #openstack-infra | 20:28 | |
| yolanda | and with plugins you just manage natively on the first layer | 20:28 |
| crinkle | greghaynes: i'll try to figure out my jira-fu to get it fixed | 20:28 |
| clarkb | yolanda: no you don't because you don't have hardware you only have what the provider provides | 20:28 |
| clarkb | yolanda: so its the same level of nesting | 20:28 |
| openstackgerrit | K Jonathan Harker proposed openstack-infra/zuul: Optionally require a zuul-ref when cloning https://review.openstack.org/227955 | 20:28 |
| yolanda | clarkb, if you extend, you could have hardware | 20:29 |
| clarkb | yolanda: sure and its the same in that case as well | 20:29 |
| clarkb | yolanda: if nova+docker does it on hardware vs nodepool talking docker its the same | 20:29 |
| *** shardy has quit IRC | 20:29 | |
| *** geoffarnold has quit IRC | 20:29 | |
| greghaynes | crinkle: so, I would still do the enroll all the things, deploy, see what 'just works' and use that | 20:29 |
| greghaynes | crinkle: just so we can get all the broken hardware in a list and get it out of the way | 20:30 |
| *** geoffarnold has joined #openstack-infra | 20:30 | |
| crinkle | greghaynes: mmk | 20:30 |
| *** dizquierdo has joined #openstack-infra | 20:30 | |
| greghaynes | crinkle: as long as we have > 10 nodes we can charge ahead on this so kind of dont want to get bogged down in fixing it all right now | 20:30 |
| *** marzif has joined #openstack-infra | 20:30 | |
| clarkb | yolanda: basically given whatever "hardware" you do have running nodepool to directly talk to pxe/ipmi/docker/etc is not really going to be different than running a nova/magnum/ironic to talk to the "hardware" instead | 20:31 |
| clarkb | yolanda: with the benefit being that they have already solved these problems and provide nice apis to take advantage of it | 20:31 |
| yolanda | clarkb so i need to find the time to test both approaches for sure. At some point i think it can be simpler to just make it flexible and extend depending on user's needs, but you are right in terms of development and reusing | 20:32 |
| *** AJaeger has quit IRC | 20:32 | |
| *** shardy_ has quit IRC | 20:33 | |
| *** shardy has joined #openstack-infra | 20:34 | |
| clarkb | yolanda: and we can work with openstack where it doesn't meet our needs | 20:34 |
| clarkb | yolanda: we have been doing that here and there (neutron get me an ip spec, glance upload feedback, keystoneauth split out, etc etc) | 20:34 |
| *** cloudtrainme has quit IRC | 20:34 | |
| *** e0ne has quit IRC | 20:35 | |
| *** cloudtrainme has joined #openstack-infra | 20:35 | |
| greghaynes | Yes, virtuous cycle | 20:36 |
| greghaynes | the angry, angry virtuous cycle | 20:36 |
| lifeless | greghaynes: so angry | 20:38 |
| lifeless | greghaynes: so virtuous | 20:38 |
| lifeless | greghaynes: btw, what about dib isn't ready for parallel image builds? | 20:38 |
| greghaynes | lifeless: No idea, I do know infra hit something I just havent had time to reproduce | 20:38 |
| clarkb | lifeless: the git repo cache stuff can't do it | 20:38 |
| clarkb | I guess thats element specific | 20:39 |
| greghaynes | There is locking around that which is supposed to work | 20:39 |
| clarkb | hrm maybe we should try it again? | 20:39 |
| clarkb | it has been a while | 20:39 |
| greghaynes | It isnt widely tested though so I am not entirely surprised you found something | 20:39 |
| lifeless | its not ci tested for sure | 20:40 |
| lifeless | but in the past I had it working happily | 20:40 |
| lifeless | we certainly are careful not to make assumptions about block device names etc | 20:40 |
| *** sambetts has quit IRC | 20:41 | |
| *** sambetts has joined #openstack-infra | 20:41 | |
| *** glauco_ has joined #openstack-infra | 20:41 | |
| *** bswartz has quit IRC | 20:42 | |
| *** matrohon has quit IRC | 20:43 | |
| *** glauco has quit IRC | 20:44 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-stackalytics: WIP: Initial commit for puppet-stackalytics https://review.openstack.org/190708 | 20:44 |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-stackalytics: WIP: Initial commit for puppet-stackalytics https://review.openstack.org/190708 | 20:45 |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-stackalytics: WIP: Initial commit for puppet-stackalytics https://review.openstack.org/190708 | 20:46 |
| *** pc_m has quit IRC | 20:46 | |
| *** DericHorn-HP has quit IRC | 20:47 | |
| *** cloudtrainme has quit IRC | 20:48 | |
| *** geoffarnold has quit IRC | 20:50 | |
| *** geoffarnold has joined #openstack-infra | 20:51 | |
| *** docaedo has quit IRC | 20:51 | |
| *** cloudtrainme has joined #openstack-infra | 20:52 | |
| clarkb | infra-root: any reason to not log nodepool.image.build.* at DEBUG level https://review.openstack.org/#/c/229718/1 ? my reading of the code is that only dib output and ssh commands for the snapshot builds will be logged at debug with this ocnfig change | 20:54 |
| clarkb | I will approve shortly if we aren't worried about leaked creds but I am pretty positive that keystone etc is isolated from this | 20:54 |
| mordred | clarkb: I think it should be fine | 20:55 |
| mordred | clarkb: also, even if it weren't, keystoneauth properly masks passwords in logging now too | 20:55 |
| greghaynes | clarkb: Yea, that should be fine | 20:55 |
| clarkb | mordred: oh thats awesome | 20:55 |
| *** docaedo_ has joined #openstack-infra | 20:56 | |
| clarkb | mordred: are we using keystoneauth everywhere yet? | 20:56 |
| fungi | clarkb: seems fine to me | 20:56 |
| mordred | clarkb: we are in nodepool | 20:56 |
| *** docaedo_ is now known as docaedo | 20:56 | |
| clarkb | mordred: oh right via occ | 20:56 |
| mordred | clarkb: yup | 20:56 |
| mordred | we're probbly the largest ksa user in the world by a very large margin at this point :) <-- jamielennox | 20:56 |
| clarkb | if we are really brave we could potentially publish the server logs too then | 20:56 |
| *** sergmelikyan has quit IRC | 20:57 | |
| greghaynes | that would make me so happy | 20:57 |
| jeblair | omg they are so big :) | 20:57 |
| jeblair | but ++ | 20:57 |
| *** thiagop has quit IRC | 20:58 | |
| clarkb | change has been approved | 20:59 |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Actually use the values provided in the conf file https://review.openstack.org/232813 | 21:00 |
| *** salv-orlando has joined #openstack-infra | 21:02 | |
| *** sridhar_ram1 has quit IRC | 21:04 | |
| *** cloudtrainme has quit IRC | 21:04 | |
| openstackgerrit | Merged openstack-infra/system-config: Turn image logging level up to DEBUG https://review.openstack.org/229718 | 21:05 |
| *** aysyd has quit IRC | 21:07 | |
| *** cloudtrainme has joined #openstack-infra | 21:07 | |
| *** sridhar_ram has joined #openstack-infra | 21:07 | |
| *** sergmelikyan has joined #openstack-infra | 21:08 | |
| mtreinish | jeblair: so how much free space do we have left on the subunit2sql db server? | 21:08 |
| *** jsavak has quit IRC | 21:12 | |
| *** geoffarnold has quit IRC | 21:12 | |
| *** mestery has quit IRC | 21:12 | |
| *** geoffarnold has joined #openstack-infra | 21:12 | |
| lifeless | jeblair: surely journalctl has a wire protocol :) | 21:14 |
| *** ldnunes has quit IRC | 21:15 | |
| *** trown is now known as trown|outttypeww | 21:15 | |
| fungi | mtreinish: says "272.7 GB of 300 GB" | 21:16 |
| mtreinish | ok, I thought we were a bit worse off than that | 21:17 |
| mtreinish | kinda takes the pressure off | 21:18 |
| *** gildub has joined #openstack-infra | 21:18 | |
| *** ianw has quit IRC | 21:18 | |
| mtreinish | fungi: fwiw, I've got the patches up to fix the expire command | 21:20 |
| fungi | mtreinish: i saw the ping over the weekend (or was it last week?) but haven't had a chance to retry | 21:20 |
| mtreinish | fungi: the patch hasn't landed yet, and I'll need to push a release once it does | 21:21 |
| fungi | oh, got it | 21:21 |
| mtreinish | https://review.openstack.org/232187 | 21:21 |
| *** ianw has joined #openstack-infra | 21:25 | |
| clarkb | mtreinish: at 272GB the disk on logstash.o.o isn't big enough to save the backups in plain text | 21:25 |
| clarkb | mtreinish: we have ~50GB to work with, not sure how small gzip will get it | 21:25 |
| mtreinish | clarkb: well the os is part of that, but it was gzipping it too | 21:26 |
| mtreinish | yeah I doubt it'll be that small after compression | 21:26 |
| * mtreinish feels like we should dump it in swift | 21:26 | |
| clarkb | mtreinish: fungi maybe we can turn on the trove backups for this db? | 21:27 |
| clarkb | mtreinish: fungi rather than relying on our backup system? that will also hit the backup servers pretty hard | 21:27 |
| mtreinish | clarkb: oh is that a thing? I didn't realize, although I'm not really surprised that's a trove feature | 21:27 |
| zaro | clarkb, jeblair : i don't see the problem you saw with this change: https://review-dev.openstack.org/#/c/5331 | 21:28 |
| clarkb | mtreinish: it is, we don't use it because they don't offer append only backups but for this DB I bet thats fine | 21:28 |
| clarkb | zaro: are you logged in? you have to log out based on discussion during meeting | 21:28 |
| zaro | i'm not logged in and it renders fine for me. both side by side and the other view | 21:28 |
| zaro | *other/inline | 21:29 |
| clarkb | zaro: https://review-dev.openstack.org/#/c/5331/2/driver.py,unified that link? | 21:29 |
| mtreinish | clarkb: yeah that would probably be fine | 21:29 |
| clarkb | zaro: it still fails for me | 21:29 |
| zaro | boom, works for me | 21:29 |
| fungi | trove backups are also not off-site like our cross-region backup solution is | 21:29 |
| fungi | but still, for that db, i'm fine with it | 21:29 |
| jeblair | clarkb, zaro: still fails for me in firefox while logged out | 21:30 |
| zaro | ohh i'm using chrome. lemme check FF | 21:30 |
| clarkb | fails in chromium too | 21:30 |
| mtreinish | fungi: yeah, I think we'll be fine with just eating a region failure causing data loss | 21:30 |
| zaro | ok. fails for me on FF | 21:30 |
| *** erikmwilson_ has joined #openstack-infra | 21:31 | |
| *** erikmwilson_ has quit IRC | 21:31 | |
| jeblair | mtreinish: do we really need backups for it? | 21:31 |
| jeblair | i mean, we don't backup logstash... | 21:31 |
| mtreinish | jeblair: it was more if we wanted to keep the data we were pruning, I wouldn't say its a requirement | 21:32 |
| clarkb | thats a good point, though logstash has N+1 redundancy | 21:32 |
| mtreinish | jeblair: also we don't have a backup solution pre-migration | 21:32 |
| clarkb | subunit2sql is a spof right now | 21:32 |
| *** annegentle has quit IRC | 21:32 | |
| mtreinish | meaning if I write a super crappy schema migration we could hose it | 21:32 |
| *** burgerk has quit IRC | 21:32 | |
| zaro | ahh, can repro on both now. thanks. | 21:33 |
| mtreinish | or I guess I should say anyone writes a bad migration, because hopefully I won't be writing everything forever | 21:33 |
| jeblair | yeah, i'm just wondering if the data has changed from "nice to have" to "development will stop without it" | 21:33 |
| clarkb | zaro: what was need in chrome? | 21:33 |
| *** geoffarn_ has joined #openstack-infra | 21:33 | |
| *** pradk has quit IRC | 21:33 | |
| zaro | arrgh! i was logged in. | 21:33 |
| mtreinish | jeblair: once the dashboard is up and running its importance does increase a bit | 21:34 |
| *** geoffarnold has quit IRC | 21:34 | |
| zaro | was confused, too many tabs going on at once. | 21:34 |
| *** nmagnezi has quit IRC | 21:34 | |
| *** erikwilson has quit IRC | 21:35 | |
| jeblair | mtreinish, clarkb: tbh, i'd prefer we make a binary choice -- either data is important and should be backed up to the best of our ability, or it is unimportant transient data and does not need to be backed up... | 21:35 |
| clarkb | jeblair: does that mean you don't find trove backups to be sufficient for this case? | 21:36 |
| jeblair | mtreinish, clarkb: i'm uncomfortable with the idea that we might consider a trove backup sufficient for something | 21:36 |
| jeblair | clarkb: indeed :) | 21:36 |
| clarkb | ok | 21:36 |
| mtreinish | jeblair: well in the past we said it was transient and it didn't matter, but I think that's starting to change. We might not be at that point yet though | 21:36 |
| jeblair | clarkb: because, tbh, "cloud deletes our account" is our primary failure mode. | 21:37 |
| mtreinish | which is why it's fuzzy | 21:37 |
| jeblair | mtreinish: okay. make sure you take that into account as you plan dashboards, etc. i don't want that change to be a surprise to anyone. | 21:37 |
| clarkb | jeblair: good point | 21:37 |
| * SpamapS is back | 21:37 | |
| SpamapS | greghaynes: done yet? | 21:37 |
| greghaynes | SpamapS: yep, cloud is done | 21:38 |
| clarkb | jeblair: in other news I am attempting to transition back to gertty now | 21:38 |
| greghaynes | SpamapS: we can all go home | 21:38 |
| jeblair | clarkb: neat! | 21:38 |
| SpamapS | greghaynes: fantastic, make two so we are redundant. thx | 21:38 |
| greghaynes | SpamapS: determined that the node is actually dead, probably some catastrophic hw issue, so yay! | 21:38 |
| mtreinish | jeblair: yeah, will do. I think that'll honestly be the turning point when we get openstack-health running in infra | 21:38 |
| clarkb | jeblair: my phone died on sunday and I have a loaner currently whih sort of makes the use webui because phone reason not super valid anymore | 21:38 |
| jeblair | clarkb: well, i'm happy and sad for you. | 21:39 |
| *** julim has quit IRC | 21:39 | |
| mordred | mtreinish, jeblair: I think it would be worthwhile for us to have a discussion about our database infrastructure at some point | 21:39 |
| jeblair | mordred: that too -- i don't want to back us into a trove corner. | 21:39 |
| greghaynes | crinkle: you ready for me to do a full redeploy? | 21:40 |
| jeblair | (though, for the moment, i'm happy with it) | 21:40 |
| mtreinish | mordred: ++ | 21:40 |
| crinkle | greghaynes: uh no | 21:40 |
| greghaynes | crinkle: :) | 21:40 |
| greghaynes | crinkle: heh | 21:40 |
| mordred | yah. at the moment I am - but there are enough places where it's just weird enough that possibly just running our own cluster might be nicer | 21:40 |
| *** eharney has quit IRC | 21:40 | |
| crinkle | greghaynes: i am pretty sure all of the hardware is le broken | 21:40 |
| greghaynes | crinkle: uhhhh | 21:40 |
| crinkle | every machine i go to has the same issue | 21:40 |
| greghaynes | crinkle: ok, let me try and boot some | 21:40 |
| crinkle | ya i just don't want to blow everything away | 21:41 |
| SpamapS | maybe they're dismantling the rack. ;) | 21:41 |
| mtreinish | mordred: I like that idea | 21:41 |
| greghaynes | crinkle: yea, a full redeploy shouldnt effect anything we care about, just basically toggles all the power states | 21:41 |
| greghaynes | crinkle: I am going to try and find a working one though | 21:42 |
| crinkle | greghaynes: toggling power is what makes it not come back | 21:42 |
| SpamapS | is pleia2 around? | 21:42 |
| *** carl_baldwin has quit IRC | 21:42 | |
| *** Piet has quit IRC | 21:42 | |
| SpamapS | I gave her the "how to open a ticket" instructions before leaving HP. | 21:42 |
| clarkb | SpamapS: today is a travel day for pleia2 iirc | 21:42 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/devstack-gate: WIP: Use devstack subunit stream in output https://review.openstack.org/234447 | 21:42 |
| SpamapS | crinkle: I think you'll want to open tickets for these. | 21:43 |
| crinkle | SpamapS: yeah i'm working on it | 21:43 |
| *** carl_baldwin has joined #openstack-infra | 21:43 | |
| SpamapS | ok cool | 21:43 |
| SpamapS | DC ops is pretty good about responding within an hour or two to non-critical tickets. | 21:43 |
| greghaynes | oh, thats nice if SpamapS actually knows how. I was only really able to tell crinkle there exists a jira and you can in theory figure out how to do that | 21:43 |
| *** shashank_hegde has quit IRC | 21:43 | |
| *** glauco has joined #openstack-infra | 21:43 | |
| crinkle | yes greghaynes is super helpful | 21:44 |
| crinkle | :P | 21:44 |
| greghaynes | always | 21:44 |
| crinkle | i have to pinpoint the right person to give me permissions to file a dcops ticket | 21:44 |
| *** sergmelikyan has quit IRC | 21:44 | |
| jeblair | clarkb, fungi, mordred: have a sec to talk about ovh? | 21:44 |
| *** dprince has quit IRC | 21:45 | |
| *** abregman|home has quit IRC | 21:46 | |
| clarkb | jeblair: yes | 21:46 |
| *** mestery has joined #openstack-infra | 21:46 | |
| clarkb | (I am mostly waiting on a gertty sync right now) | 21:46 |
| *** gordc has quit IRC | 21:47 | |
| *** glauco_ has quit IRC | 21:47 | |
| greghaynes | crinkle: just tried 98d141a3-7efc-4df3-84a6-e3d8cdff3025 - seems to work totally fine | 21:47 |
| *** rfolco has quit IRC | 21:48 | |
| jeblair | clarkb, fungi, mordred: (hopefully others will catch up) | 21:48 |
| jeblair | clarkb, fungi, mordred: http://paste.openstack.org/show/476195/ | 21:48 |
| mordred | yup | 21:48 |
| *** mriedem has quit IRC | 21:48 | |
| crinkle | greghaynes: well you must have better luck than me | 21:48 |
| jeblair | clarkb, fungi, mordred: i've managed to run those 2 jobs approx 10 times each | 21:48 |
| greghaynes | crinkle: yea. It is very possible more than 1 node has died since we started - this is not new hw by a long shot | 21:48 |
| mordred | jeblair: so those are time-in-seconds? | 21:49 |
| jeblair | clarkb, fungi, mordred: (the hpcloud and rax data are from about a month of runs in nodepool; ovh is me running approx 10 times manually) | 21:49 |
| jeblair | yep | 21:49 |
| jeblair | clarkb, fungi, mordred: hpcloud is approx 1.3 * rax; ovh is approx 1.7-1.8 * rax | 21:49 |
| clarkb | jeblair: and thats the average? | 21:49 |
| jeblair | clarkb: yes, mean | 21:49 |
| mordred | jeblair: hrm. that's sad panda | 21:49 |
| clarkb | jeblair: that isn't super surprising considering we have half the cpu count | 21:50 |
| jeblair | clarkb, fungi, mordred: yeah. i think that means we should ask them for 8vcpu instead of 4. | 21:50 |
| clarkb | jeblair: is there an existing flavor we can use really quickly to just test with? | 21:50 |
| clarkb | jeblair: so confirm it before we go asking? | 21:51 |
| fungi | i am catching up. in the middle of cooking dinner and need to not burn things though | 21:51 |
| mtreinish | jeblair: this is the graph I made last week for you with similar things being measured, I can work on doing it split per region now: http://i.imgur.com/VQ5RRWK.png | 21:51 |
| mordred | clarkb: I believe we're using a flavor they made special for us | 21:51 |
| mtreinish | forgot about that before | 21:51 |
| clarkb | mordred: we are, but I hope we can use other flavors too? | 21:51 |
| jeblair | mtreinish: cool, i have the outputs from the runs, so i can double check the values from tempest | 21:52 |
| clarkb | I am just thinking it would be good to get something similar to confirm before we ask them to update the special flavor | 21:52 |
| anteaya | fungi: not burning things is good | 21:52 |
| * anteaya also needs to not burn things | 21:52 | |
| jeblair | clarkb: maybe we tell them we think we need 8, and ask whether they have a flavor ready we can use to confirm before they make the change, or if they want to make the change? | 21:52 |
| mordred | clarkb: we CAN - but the other flavors do not have the appropriate amount of disk we'd want | 21:52 |
| clarkb | jeblair: that sounds reasonable too | 21:52 |
| jeblair | mordred: to be fair, i think they have more? | 21:53 |
| mordred | jeblair: possibly? | 21:53 |
| jeblair | clarkb, fungi, mordred: that's thing 1... i have 2 more things: | 21:53 |
| mordred | jeblair: ooh. things! | 21:53 |
| jeblair | clarkb, fungi, mordred: thing 2 and 3 are related -- these hosts have no swap, and they have 40G of disk (all root, no ephemeral). | 21:54 |
| *** flepied has joined #openstack-infra | 21:54 | |
| *** geoffarn_ has quit IRC | 21:54 | |
| jeblair | clarkb, fungi, mordred: elsewhere, we turn part of the ephemeral into swap and use the rest for tests | 21:54 |
| *** geoffarnold has joined #openstack-infra | 21:55 | |
| jeblair | clarkb, fungi, mordred: strictly speaking, that doesn't appear to be needed for a regular single-node devstack run, which is sort of our standard | 21:55 |
| *** claudiub has quit IRC | 21:55 | |
| *** amuller has quit IRC | 21:55 | |
| *** Sukhdev_ has joined #openstack-infra | 21:55 | |
| jeblair | clarkb, fungi, mordred: though, theoretically, the lack of swap could potentially be reflected in a longer execution time | 21:55 |
| *** sabeen1 has joined #openstack-infra | 21:55 | |
| jeblair | clarkb, fungi, mordred: however, i don't think we actually use swap that much. looking at dstat for an hpcloud/rax run, it appears to swap only a few times, and probably about 50MB total | 21:56 |
| mordred | nod | 21:56 |
| clarkb | jeblair: ya Ithink after the worker retuning post PyMySQL switch the swap usage became less crazy | 21:56 |
| jeblair | clarkb, fungi, mordred: we know that in some tests we're using cinder with lvm, but i've proposed a revert of that here: https://review.openstack.org/232132 | 21:57 |
| jeblair | clarkb, fungi, mordred: using cinder with lvm requires an ephemeral volume | 21:57 |
| clarkb | I was hoping jgriffith would weigh in on that one | 21:57 |
| clarkb | since jgriffith was part of the original switch to the non file backed lvm | 21:58 |
| jeblair | clarkb, fungi, mordred: and we know that there are some jobs that use more space than devstack... | 21:58 |
| jgriffith | clarkb: I'm mixed | 21:58 |
| jeblair | clarkb: do you recall the project we saw run out of space? | 21:58 |
| clarkb | jeblair: it was openstack ansible | 21:58 |
| jgriffith | clarkb: I think what actually fixed us at that time was the lvm conf | 21:58 |
| *** dimtruck is now known as zz_dimtruck | 21:58 | |
| jgriffith | clarkb: I *believe* we are ok to remove that... I'll mark my vote as such on the review | 21:58 |
| clarkb | jeblair: their testing is spin up lots of ocntainers in one VM and do things | 21:58 |
| mordred | jeblair: I think the openstack-ansible thing could likely be fixed - I believe they're doing more work in cloning and copying than is needed | 21:58 |
| clarkb | jeblair: its possible multinode is a better map for them | 21:59 |
| *** sergmelikyan has joined #openstack-infra | 21:59 | |
| *** erikwilson has joined #openstack-infra | 21:59 | |
| clarkb | also ya they could likely reduce their disk usage significantly | 21:59 |
| mordred | and they could make use of stuff we have there already better I think | 21:59 |
| jeblair | clarkb, fungi, mordred: so at the risk of being exceedingly meta -- how should we go about deciding whether we need swap, whether we need ephemeral, and how much total disk space we need? | 22:00 |
| fungi | right, i think they sized their disk usage based on the smallest filesystem on the providers we had | 22:00 |
| greghaynes | crinkle: just had same luck with b9b47664-c5d0-472c-b232-8c1a4ef22fd0 | 22:00 |
| greghaynes | crinkle: so I think things arent abnormally bad | 22:00 |
| jeblair | fungi: yeah, and we didn't so much anticipate the lack of ephemeral | 22:00 |
| mordred | jeblair: so ... one of the reasons that we pin to 8G | 22:01 |
| SpamapS | greghaynes: it's also possible these are known bad and we're just getting our inventory from before mixed up | 22:01 |
| SpamapS | crinkle: ^ | 22:01 |
| *** glauco has quit IRC | 22:01 | |
| mordred | is that 8G is a reasonable amount of memory for someone to have on their laptop | 22:01 |
| jeblair | mordred: ya | 22:01 |
| mordred | so that they could be reasonably expected to be able to run things locally | 22:01 |
| crinkle | SpamapS: there is at least one that I definitely know was working and suddenly stopped | 22:01 |
| *** glauco has joined #openstack-infra | 22:01 | |
| greghaynes | SpamapS: Yea. So I think next step is do a full redeploy with stock image, see what boots, unrell everything that doesnt and just work with the known good list for now | 22:01 |
| mordred | I think I'd start from there | 22:01 |
| greghaynes | crinkle: Yea, I think -24 is one I was using before too | 22:01 |
| mordred | or else I think we should also refigure RAM if we use a different set of criteria | 22:01 |
| greghaynes | SpamapS: er, unenroll | 22:02 |
| fungi | i'm inclined to just assume we don't need swap. if a job _really_ needs it, they have root access and you can mkswap and use a file on the filesystem | 22:02 |
| jeblair | mordred: true, though also, if we increase our ram footprint we sacrifice quota (potentially half of it) so it's not the only reason :) | 22:02 |
| greghaynes | crinkle: SpamapS ok, I have to run for an hourish. | 22:02 |
| fungi | *supposedly* swap files on most modern linux filesystem implementations perform nearly as well as dedicated swap block devices | 22:03 |
| *** erikmwilson_ has joined #openstack-infra | 22:03 | |
| mordred | jeblair: totally | 22:03 |
| crinkle | greghaynes: aw i was gonna say please go ahead and redeploy | 22:03 |
| *** erikwilson has quit IRC | 22:03 | |
| greghaynes | crinkle: well, you can go for it, otherwise I can do it in a bit | 22:03 |
| greghaynes | or SpamapS can | 22:03 |
| *** glauco__ has joined #openstack-infra | 22:03 | |
| jeblair | fungi, clarkb, mordred: i think 40G is essentially our floor now, but i think we need to check grenade | 22:03 |
| *** sabeen3 has joined #openstack-infra | 22:04 | |
| crinkle | i will try it out, hopefully my bad luck doesn't affect us too much | 22:04 |
| mordred | jeblair: if that's our floor and we wind up being ok with that - should we change our partitioning on the other nodes like we ram limit on hp? | 22:04 |
| jeblair | fungi, clarkb, mordred: i'm inclined to, but unfortunately, i think hpcloud gives us 30 root + ephemeral, so i think we'd still need to use ephemeral for /opt there | 22:04 |
| *** sabeen1 has quit IRC | 22:04 | |
| clarkb | jeblair: I think rax is 30GB too? | 22:05 |
| clarkb | but yes I agree we should try to keep them from diverging too much to avoid random "this job failed and I don't know why" problems | 22:05 |
| jeblair | clarkb: oh? | 22:05 |
| *** asettle has joined #openstack-infra | 22:05 | |
| jeblair | okay, so i think i'm seeing the next steps here.. let me see if i can summarize | 22:05 |
| *** ybathia has joined #openstack-infra | 22:05 | |
| *** glauco has quit IRC | 22:06 | |
| *** dramalho_ has quit IRC | 22:06 | |
| *** regXboi has quit IRC | 22:06 | |
| *** rossella_s has quit IRC | 22:06 | |
| jeblair | clarkb, fungi, mordred: 1) we have decided to stop making swap: we'll remove that from d-g and add it to our specifications. 2) i need to check the disk usage for / and /opt for regular devstack and grenade jobs, and see if i can define a requirement and come up with a partitioning scheme to match and unify providers | 22:07 |
| *** rossella_s has joined #openstack-infra | 22:07 | |
| jeblair | the results of 2) will inform what we say about root and ephemeral volumes | 22:07 |
| *** ybathia has quit IRC | 22:07 | |
| *** denisra has joined #openstack-infra | 22:07 | |
| clarkb | jeblair: sounds good | 22:07 |
| mordred | ++ | 22:08 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 22:08 | |
| jeblair | clarkb, fungi, mordred: how about i write back to ovh and say "we think we need more procs; can you suggest/make a flavor so we can double check. we think we may be close on disk space, we'll get that info asap." ? | 22:08 |
| clarkb | +1 | 22:08 |
| *** sergmelikyan has quit IRC | 22:09 | |
| *** amitgandhinz has quit IRC | 22:09 | |
| *** david-lyle has quit IRC | 22:10 | |
| jeblair | thanks! | 22:10 |
| *** dra_ has quit IRC | 22:10 | |
| mordred | ++ | 22:10 |
| fungi | jeblair: lgtm | 22:10 |
| fungi | thanks! | 22:10 |
| mordred | jeblair: would it be helpful for me to do anything with the BB cloud? or should I leave that on your plate since you've been doing the OVH one? | 22:10 |
| *** glauco__ has quit IRC | 22:11 | |
| *** david-lyle has joined #openstack-infra | 22:12 | |
| jeblair | mordred: i think i can still multiplex it, thanks. | 22:13 |
| *** tonytan4ever has quit IRC | 22:13 | |
| mordred | jeblair: kk | 22:13 |
| jeblair | can probably start in earnest tomorrow | 22:13 |
| *** gondoi is now known as zz_gondoi | 22:13 | |
| clarkb | greghaynes: why do you need a second lock in https://review.openstack.org/#/c/195175/21..29/nodepool/builder.py ? if you use the start lock in the outer block aren't you already exclusive? | 22:14 |
| *** tongli has quit IRC | 22:15 | |
| openstackgerrit | Hamdi Roumani proposed openstack-infra/project-config: Add #storlets to gerritbot https://review.openstack.org/234350 | 22:15 |
| *** geoffarnold has quit IRC | 22:15 | |
| *** shardy_ has joined #openstack-infra | 22:15 | |
| *** geoffarnold has joined #openstack-infra | 22:16 | |
| openstackgerrit | Hamdi Roumani proposed openstack-infra/project-config: Add #openstack-storlets to gerritbot https://review.openstack.org/234350 | 22:16 |
| *** shardy has quit IRC | 22:17 | |
| jeblair | clarkb, fungi, mordred: i forgot to mention also that i have attempted to run multinode tests, but they failed and i have not examined why. | 22:18 |
| jeblair | that might have an impact too. | 22:18 |
| clarkb | jeblair: were you able to see if it was devstack+devstack-gate that failed or tempest? | 22:19 |
| *** xyang1 has quit IRC | 22:19 | |
| jeblair | clarkb: d+d-g | 22:19 |
| *** ybathia has joined #openstack-infra | 22:20 | |
| clarkb | interesting | 22:20 |
| *** mestery has quit IRC | 22:21 | |
| jeblair | clarkb: http://paste.openstack.org/show/476196/ | 22:21 |
| anteaya | Clint: for reference: http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2015-10-13.log.html#t2015-10-13T20:10:17 | 22:21 |
| jeblair | clarkb: is the tail of the log | 22:21 |
| *** shardy_ has quit IRC | 22:21 | |
| Clint | anteaya: right | 22:21 |
| anteaya | Clint: thanks | 22:21 |
| jeblair | clarkb: i'm working on reproducing that in a more accessible manner next, i might ask you to hop on-host at some point :) | 22:21 |
| clarkb | jeblair: ok | 22:21 |
| *** shardy has joined #openstack-infra | 22:22 | |
| *** annegentle has joined #openstack-infra | 22:22 | |
| *** ybathia has quit IRC | 22:22 | |
| anteaya | mtreinish: can I bug you to look at a yaml2ical patch for a review please? https://review.openstack.org/#/c/232312/ | 22:26 |
| anteaya | mtreinish: it adds the feature to be able to specify a timestamp to skip a meeting date | 22:26 |
| anteaya | I would like it soon so I can skip the summit dates in the ical feed | 22:27 |
| anteaya | mtreinish: you know yaml-y things | 22:27 |
| *** cody-somerville has quit IRC | 22:27 | |
| anteaya | mtreinish: or at least you have left me with that impression | 22:27 |
| clarkb | greghaynes: I commented you can refresh to see comment | 22:28 |
| *** erikmwilson_ has quit IRC | 22:28 | |
| greghaynes | clarkb: I will just have you show me at your place in 30ish :p | 22:29 |
| clarkb | ok | 22:29 |
| greghaynes | because phone is not so fun for code review | 22:29 |
| *** cody-somerville has joined #openstack-infra | 22:29 | |
| openstackgerrit | Sahdev Zala proposed openstack-infra/irc-meetings: Add Heat-Translator team meeting https://review.openstack.org/234458 | 22:30 |
| *** EricGonc_ has quit IRC | 22:31 | |
| jesusaurus | i cant figure out why https://review.openstack.org/#/c/233801/ claims there's a merge conflict, it seems up-to-date with gerrit/master on my end | 22:33 |
| *** edmondsw has quit IRC | 22:33 | |
| clarkb | dhellmann laying down the law with his first and only warning | 22:33 |
| jesusaurus | could someone take a peek at the zuul-merger log to see if its failing in an unusual way? | 22:33 |
| clarkb | jesusaurus: that was around the time of zuul mergers running out of inodes | 22:34 |
| *** sabeen3 has quit IRC | 22:34 | |
| clarkb | jesusaurus: I would attempt a recheck | 22:34 |
| jesusaurus | ah, ok | 22:34 |
| *** darrenc is now known as darrenc_afk | 22:35 | |
| armax | clarkb: I don’t see Andreas online, http://docs.openstack.org/developer/neutron/policies/index.html went automagically back to being the latest version | 22:36 |
| *** geoffarnold has quit IRC | 22:37 | |
| armax | clarkb: who should I thank? | 22:37 |
| armax | :) | 22:37 |
| zaro | fungi, clarkb, jeblair, anteaya, pleia2 : draft email RE gerrit upgrade: https://etherpad.openstack.org/p/announce-gerrit-upgrade | 22:37 |
| *** sabeen1 has joined #openstack-infra | 22:37 | |
| clarkb | armax: uh I don't know I missed most of that due to our meeitng | 22:37 |
| *** geoffarnold has joined #openstack-infra | 22:37 | |
| anteaya | mordred: ^^ | 22:37 |
| jroll | armax: heh, we noticed that the stable kilo release blew away ironic docs too | 22:37 |
| armax | jroll: ah | 22:37 |
| anteaya | armax: random thanks are fine | 22:38 |
| armax | thanks infra channel! | 22:38 |
| jroll | armax: we figured the next docs patch would fix, seems like you've verified that :) | 22:38 |
| jroll | I don't know that it's actually fixed for real but hey | 22:38 |
| armax | jroll: which patch? | 22:38 |
| anteaya | armax: welcome | 22:38 |
| crinkle | greghaynes: i don't believe you about b9b47664-c5d0-472c-b232-8c1a4ef22fd0 | 22:38 |
| mtreinish | anteaya: sure I'll take a look | 22:38 |
| anteaya | mtreinish: thanks | 22:38 |
| anteaya | armax: don't look a gift bug in the mouth | 22:38 |
| mtreinish | anteaya: I'm heads down trying to generate a fancy graph, but after I get that working | 22:38 |
| crinkle | greghaynes: or i apparently have the midas touch | 22:38 |
| *** thorst has quit IRC | 22:39 | |
| anteaya | mtreinish: ooh will there be labels? | 22:39 |
| armax | anteaya: :) | 22:39 |
| crinkle | greghaynes: either way i'm not gonna redeploy everything | 22:39 |
| jroll | armax: what I mean is; we think doing the kilo ironic point release generated a docs publish, and pushed kilo docs to docs.o.o. and then the next ironic docs patch would publish master docs again | 22:39 |
| * anteaya likes mtreinish's fancy graphs with labels | 22:39 | |
| anteaya | armax: :) | 22:39 |
| mtreinish | anteaya: hopefully not a label explosion this time | 22:39 |
| jroll | armax: so I assume some neutron patch landed and did a docs publish | 22:39 |
| armax | jroll: I see | 22:39 |
| mtreinish | if I have any good ones I will share | 22:39 |
| armax | jroll: I didn’t see one land recently | 22:39 |
| anteaya | mtreinish: I _like_ the label explosion | 22:39 |
| anteaya | mtreinish: thanks | 22:39 |
| armax | jroll: so this means we may be flpping between releases? :) | 22:40 |
| jroll | armax: oh, hm. dunno then :) | 22:40 |
| *** erikwilson has joined #openstack-infra | 22:40 | |
| clarkb | armax: jroll I can look at logs and jobs to see if thats the case | 22:40 |
| clarkb | I thought stable releases should publish to /developer/project/releasename | 22:41 |
| *** marzif has quit IRC | 22:41 | |
| clarkb | but that may be just for clients, it has been a couple years | 22:41 |
| fungi | zaro: lgtm. i updated the subject line to add the infra tag and drop the openstack-dev tag (that second one gets added automatically by the mailing list software anyway) | 22:41 |
| *** marzif has joined #openstack-infra | 22:41 | |
| jroll | clarkb: yeah, these are just wild assumptions we made based on the timeline | 22:41 |
| armax | clarkb, jroll: I’ll keep an eye…I wonder if there is something that we do wrong in our doc tree/config | 22:41 |
| anteaya | zaro: well written, I changes the verb tense from past to present tense | 22:42 |
| armax | clarkb: actually what I noticed, and that baffles me, is that the doc job doesn’t fails on errors | 22:42 |
| armax | even though pbr is configured outherwise | 22:42 |
| anteaya | I changed | 22:42 |
| armax | https://github.com/openstack/neutron/blob/master/setup.cfg#L200 | 22:42 |
| armax | so if I do: rm -rf doc/build -rf; python setup.py build_sphinx it fails, but if I do: rm -rf doc/build -rf; tox -r -evenv -- python setup.py build_sphinx it doesn’t | 22:43 |
| armax | the latter command is what the docs job invokes | 22:43 |
| armax | so right now I am scratching my head a bit | 22:43 |
| clarkb | armax: have a link to where it fails? | 22:44 |
| armax | clarkb: this one is an example: https://jenkins05.openstack.org/job/gate-neutron-docs/8596/console | 22:45 |
| armax | clarkb: I see the error around 2015-10-13 19:11:03.955 | 22:45 |
| armax | but then the build doesn’t fail | 22:45 |
| clarkb | ok confirmed that the run-docs.sh script attempts to write the stable branch docs to /developer/project/releasename | 22:46 |
| anteaya | zaro: did you want more feedback that what fungi and myself have provided? | 22:46 |
| mtreinish | infra-root: can someone do me a favor and check the subunit worker logs on subunit-worker01.o.o, I think it's stuck/dead because there's no more space | 22:46 |
| clarkb | armax: that build succeeded according to the log | 22:46 |
| zaro | anteaya: now sure if anybody wants to provide feedback | 22:46 |
| mtreinish | no new events have been added to the db since the 8th | 22:46 |
| clarkb | armax: there is a warning though | 22:46 |
| zaro | anyone else i mean | 22:46 |
| anteaya | zaro: I say ship it, then we can start the conversation | 22:46 |
| clarkb | armax: is the warning what you are referring to as should be causing a failure? | 22:46 |
| anteaya | zaro: I think you have covered the main points | 22:46 |
| anteaya | fungi: do you disagree? | 22:47 |
| armax | clarkb: I would expect the build to fail because I am telling pbr to treat warning as errors | 22:47 |
| armax | clarkb: here: https://github.com/openstack/neutron/blob/master/setup.cfg#L200 | 22:47 |
| *** ybathia has joined #openstack-infra | 22:47 | |
| armax | clarkb: so when I ran the command outside the venv | 22:47 |
| anteaya | zaro: fungi has left the etherpad | 22:47 |
| zaro | anteaya: maybe wait a few hours? | 22:47 |
| zaro | 1hr? | 22:48 |
| armax | clarkb: I get a sphinx.errors.SphinxWarning that breaks the build | 22:48 |
| armax | clarkb: which is what I would expect | 22:48 |
| anteaya | zaro: sure, I can support a 1 hour wait | 22:48 |
| *** skylerberg has joined #openstack-infra | 22:49 | |
| jeblair | mtreinish, infra-root: i can check | 22:49 |
| jeblair | mtreinish: why do you think there's no more space? | 22:49 |
| nibalizer | mtreinish: there is recent activity in the log, and the filesystems don't look full | 22:49 |
| jeblair | mtreinish: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=2388&rra_id=all | 22:49 |
| mtreinish | jeblair: there hasn't been any new runs written to the db since the 8th | 22:49 |
| mtreinish | jeblair, nibalizer: http://paste.openstack.org/show/476197/ | 22:50 |
| jeblair | nibalizer: i'll let you take it from here; let me know if you need anything | 22:50 |
| jeblair | mordred: there are lots of rally users in our bluebox cloud... what's up with that? | 22:51 |
| mtreinish | nibalizer: are there errors in the log or anything? | 22:51 |
| jeblair | mordred: (and projects) | 22:51 |
| nibalizer | jeblair: sure | 22:51 |
| *** erikwilson has quit IRC | 22:51 | |
| SpamapS | jeblair: sounds like somebody aborted a rally run | 22:51 |
| clarkb | armax: pbr hardcodes warningiserror=False when calling sphinx so now I have no idea how that ever works | 22:52 |
| openstackgerrit | Clint Adams proposed openstack-infra/project-config: WIP - testing the rename scripts https://review.openstack.org/229601 | 22:52 |
| armax | clarkb: interesting | 22:53 |
| nibalizer | mtreinish: http://paste.openstack.org/show/476198/ | 22:53 |
| armax | clarkb: so I was looking at https://github.com/openstack/nova/commit/a507d42cf5d9912c2b3622e84afb8b7d3278595b and the nova docs job | 22:53 |
| lifeless | clarkb: there's a weirdness on newer sphinxes IIRC | 22:53 |
| armax | clarkb: and I couldn’t see any errors being emitted | 22:54 |
| lifeless | basically the lack of testing around this has bitten us hard | 22:54 |
| boris-42 | SpamapS: jeblair guys I am working on handling SIGINT | 22:54 |
| nibalizer | oh wait here we go | 22:54 |
| boris-42 | SpamapS: it will be merged on this week https://review.openstack.org/#/c/234031/ | 22:54 |
| boris-42 | SpamapS: and I will cut new release with this | 22:55 |
| mtreinish | nibalizer: hmm yeah that work_exception state being reported back to gearman means something died | 22:55 |
| jeblair | mordred, SpamapS: also, it says there is a 1 vcpu instance on one of the hypervisors, but i can't seem to find it anywhere. | 22:55 |
| lifeless | armax: clarkb: https://bugs.launchpad.net/pbr/+bug/1496882 | 22:55 |
| mtreinish | or at least I think it does | 22:55 |
| openstack | Launchpad bug 1496882 in PBR "warnerrors = True no more work with Sphinx < 1.3" [Critical,In progress] - Assigned to Doug Hellmann (doug-hellmann) | 22:55 |
| nibalizer | too many open files | 22:55 |
| nibalizer | http://paste.openstack.org/show/476199/ | 22:55 |
| armax | lifeless: that’s the bunny I was looking for! | 22:55 |
| SpamapS | boris-42: good to hear. I love well behaved programs. :) | 22:55 |
| clarkb | jroll: the most recent ironic stable/liberty change did the correct copying thing http://logs.openstack.org/9f/9f9022f159a013ee4b14991742807bbca3bd3435/post/ironic-docs/2c2fb4e/console.html.gz#_2015-10-07_02_01_19_687 | 22:56 |
| clarkb | jroll: oh! | 22:56 |
| *** flepied has quit IRC | 22:56 | |
| mtreinish | nibalizer: hmm, I don't think we've hit that before | 22:56 |
| clarkb | I know what the problem is | 22:56 |
| clarkb | patch incoming | 22:56 |
| jroll | clarkb: heh | 22:56 |
| mtreinish | clarkb to the rescue | 22:56 |
| armax | clarkb: superman | 22:57 |
| boris-42 | SpamapS: btw why you don't use existing users for benchmarking? | 22:57 |
| SpamapS | boris-42: I did not run Rally | 22:57 |
| SpamapS | boris-42: just have seen that effect before. | 22:57 |
| mtreinish | nibalizer: is the subprocess not closing properly? | 22:57 |
| * mtreinish checks the worker script again | 22:57 | |
| SpamapS | boris-42: especially when scale testing keystone itself. | 22:57 |
| armax | lifeless: so am I right into assuming that this is affecting projects across the board? | 22:57 |
| openstackgerrit | Clint Adams proposed openstack-infra/project-config: Add script to facilitate project renaming in {projects,channels,layout}.yaml https://review.openstack.org/228002 | 22:58 |
| *** xnox has quit IRC | 22:58 | |
| boris-42 | SpamapS: heh yep if you kill it in the middle mad things will happen=( | 22:58 |
| clarkb | jroll: did ironic get a tag recently? | 22:58 |
| jroll | clarkb: no, unless that kilo release earlier today produced a tag | 22:58 |
| jroll | it probably did | 22:58 |
| * jroll looks | 22:58 | |
| nibalizer | mtreinish: I'm not sure... using lsof it looks like there are a lot of pipes that are hanging around | 22:58 |
| armax | lifeless: as far as I can tell we’re relying on the global pin for sphinx | 22:59 |
| jroll | clarkb: yeah 2015.1.2 5 hours ago | 22:59 |
| nibalizer | I've never looked at pipes with lsof before though so I'm not really sure how to interpret this data | 22:59 |
| *** geoffarn_ has joined #openstack-infra | 22:59 | |
| anteaya | Clint: thank you | 22:59 |
| *** chlong has quit IRC | 22:59 | |
| clarkb | I may not actually write a patch because what is going on is :( | 22:59 |
| mtreinish | nibalizer: yeah that's probably it, looking at the script we don't need the subprocess call anymore either: http://git.openstack.org/cgit/openstack-infra/puppet-subunit2sql/tree/files/subunit-gearman-worker.py#n96 | 22:59 |
| clarkb | but I will explain | 22:59 |
| mtreinish | we don't store subunit v1 on the disk anymore | 22:59 |
| Clint | anteaya: yup | 22:59 |
| mtreinish | so there's no reason to try and convert it | 23:00 |
| mtreinish | nibalizer: I'll push a patch one sec | 23:00 |
| nibalizer | we have about 3k of these http://paste.openstack.org/show/476200/ | 23:00 |
| nibalizer | mtreinish: sweet | 23:00 |
| lifeless | armax: yes | 23:00 |
| lifeless | armax: there's a draft patch for pbr | 23:00 |
| clarkb | jroll: https://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/scripts/run-docs.sh#n47 totally doesn't handle the tag shcema switch | 23:00 |
| *** sambetts has quit IRC | 23:00 | |
| *** xnox has joined #openstack-infra | 23:00 | |
| lifeless | armax: I haven't looked at it recently | 23:00 |
| clarkb | jroll: so the kilo tag was treated as latest | 23:00 |
| armax | lifeless: I just voted on it | 23:01 |
| armax | lifeless: https://review.openstack.org/#/c/229951/ | 23:01 |
| clarkb | jroll: and we published to the project root | 23:01 |
| jroll | clarkb: oh, because 2015.x > 4.x? | 23:01 |
| clarkb | jroll: ya | 23:01 |
| jroll | ya | 23:01 |
| *** otter768 has quit IRC | 23:01 | |
| jroll | dang | 23:01 |
| clarkb | so the simple fix would be to let post publish to root | 23:01 |
| *** xnox has quit IRC | 23:01 | |
| *** otter768 has joined #openstack-infra | 23:01 | |
| jroll | clarkb: post does publish to root | 23:01 |
| clarkb | and only publish to the tag specific things with tags | 23:01 |
| clarkb | jroll: ya | 23:01 |
| jroll | just need to trigger it | 23:01 |
| jroll | ok | 23:02 |
| clarkb | but splitting it that way should be fine I think | 23:02 |
| jroll | nod | 23:02 |
| *** xnox has joined #openstack-infra | 23:02 | |
| greghaynes | crinkle: I only verified it got past post, anything past that I assume is our failings | 23:02 |
| *** sambetts has joined #openstack-infra | 23:02 | |
| *** ZZelle_ has quit IRC | 23:02 | |
| *** geoffarnold has quit IRC | 23:03 | |
| crinkle | greghaynes: it doesn't power on anymore | 23:03 |
| *** carl_baldwin has quit IRC | 23:04 | |
| anteaya | mtreinish: I have to go offline, please help me in my plight to be able to communicate to third party operators that I won't be chairing meetings during summit | 23:04 |
| anteaya | mtreinish: thank you | 23:04 |
| greghaynes | crinkle: oh, what | 23:06 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/puppet-subunit2sql: Remove subunit v1 to v2 conversion step https://review.openstack.org/234465 | 23:06 |
| *** xnox has quit IRC | 23:07 | |
| mtreinish | clarkb, nibalizer: ^^^ | 23:07 |
| *** dramalho has joined #openstack-infra | 23:07 | |
| *** sambetts has quit IRC | 23:07 | |
| *** gyee has joined #openstack-infra | 23:07 | |
| clarkb | jroll: actually we can use the same hack that the cleints use | 23:08 |
| *** Daisy has joined #openstack-infra | 23:08 | |
| clarkb | jroll: I will write patch afterall | 23:09 |
| *** Piet has joined #openstack-infra | 23:10 | |
| jroll | clarkb: <3 | 23:10 |
| nibalizer | mtreinish: why is the f.close() line getting removed there? | 23:11 |
| mtreinish | nibalizer: because I stopped using f as an intermediate to pass to the subunit_1_to_2 method and it just became buf | 23:12 |
| *** dramalho has quit IRC | 23:12 | |
| *** Daisy has quit IRC | 23:12 | |
| mtreinish | I don't think I can close buf before it get's read by subunit2sql in the event processor | 23:12 |
| *** moravec has quit IRC | 23:13 | |
| *** MarkAtwood has quit IRC | 23:13 | |
| mtreinish | nibalizer: I might still be leaking files though, I didn't really check it thorougly enough I guess | 23:13 |
| greghaynes | crinkle: AWESOME | 23:13 |
| nibalizer | oh i see | 23:14 |
| crinkle | greghaynes: (╯°□°）╯︵ ┻━┻ | 23:14 |
| greghaynes | crinkle: so yea, I would support waiting until DC ops can say something, it seems odd how many are like that :/ | 23:14 |
| greghaynes | I hope its not a "your hardware is old and all dying" | 23:15 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Handle new release versions in doc publishing https://review.openstack.org/234467 | 23:15 |
| crinkle | greghaynes: i've pung some people | 23:15 |
| *** sabeen1 has quit IRC | 23:15 | |
| clarkb | jroll: armax ^ that should make things better again | 23:16 |
| armax | lifeless: fix https://review.openstack.org/#/c/229951/ works for me, but I wonder if we should pull the flag as specified in setup.cfg | 23:16 |
| * armax looks | 23:16 | |
| *** cloudtrainme has quit IRC | 23:17 | |
| armax | clarkb: btw I figured the issue with my sphinx | 23:17 |
| armax | my venv had pbr 1.8 | 23:17 |
| greghaynes | crinkle: ok, awesome | 23:17 |
| armax | whereas my box has pbr 1.6 | 23:17 |
| armax | and the flags were misaligned, hence it was silently failing in the venv | 23:18 |
| clarkb | ah | 23:18 |
| *** erikwilson has joined #openstack-infra | 23:18 | |
| armax | rookie mistakes | 23:18 |
| nibalizer | mtreinish: +2 from me | 23:18 |
| nibalizer | maybe clarkb wants to look at it too? | 23:18 |
| clarkb | aroo | 23:18 |
| *** geoffarn_ has quit IRC | 23:18 | |
| armax | clarkb: that’s the pbr fix to reinstate the original behavior https://review.openstack.org/#/c/229951/ | 23:19 |
| *** salv-orlando has quit IRC | 23:19 | |
| mtreinish | nibalizer: actually just traced through the code we never close subunit_v2 from the event processor either | 23:19 |
| *** salv-orlando has joined #openstack-infra | 23:20 | |
| mtreinish | so it probably makes sense to do subunit_v2.close() after we populate the db | 23:20 |
| mtreinish | I can respin that patch, or push it as a followup | 23:20 |
| *** geoffarnold has joined #openstack-infra | 23:20 | |
| mtreinish | err, follow up it is | 23:20 |
| *** flepied has joined #openstack-infra | 23:20 | |
| SpamapS | mtreinish: question on the subunit gearman worker | 23:21 |
| SpamapS | mtreinish: what submits the push-subunit job? | 23:21 |
| mtreinish | SpamapS: the logstash-gearman-client: git.openstack.org/cgit/openstack-infra/puppet-log_processor/tree/files/log-gearman-client.py | 23:22 |
| SpamapS | mtreinish: ahh, didn't have that one checked out to grep :) | 23:23 |
| *** stevemar_ has quit IRC | 23:23 | |
| *** stevemar_ has joined #openstack-infra | 23:24 | |
| mtreinish | SpamapS: the config file we use for that is: git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/files/logstash/jenkins-log-client.yaml | 23:25 |
| mtreinish | err, why is firefox dropping https:// now when I copy links | 23:25 |
| SpamapS | mtreinish: indeed, pieces falling into place in head now | 23:25 |
| SpamapS | though now I wonder where the other side of zmq is | 23:26 |
| nibalizer | SpamapS: this may help http://docs.openstack.org/infra/system-config/logstash.html | 23:26 |
| mtreinish | SpamapS: jenkins somethingy hand waving | 23:26 |
| nibalizer | clark (i think) made an ascii diagram | 23:26 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/puppet-subunit2sql: Ensure we close the subunit_v2 stream file object https://review.openstack.org/234469 | 23:26 |
| mtreinish | clarkb, nibalizer: ^^^ part 2 | 23:26 |
| SpamapS | nibalizer: thats the ticket | 23:26 |
| *** salv-orlando has quit IRC | 23:27 | |
| mtreinish | nibalizer: yep clarkb made it and I added the clearly added on piece for subunit | 23:27 |
| *** Rockyg has quit IRC | 23:27 | |
| *** erikwilson has quit IRC | 23:27 | |
| nibalizer | haha | 23:27 |
| *** erikwilson has joined #openstack-infra | 23:28 | |
| mtreinish | SpamapS: or if you like colors and shapes I made one with those too: http://blog.kortar.org/wp-content/uploads/2015/08/flowchart.png (which I based on another clarkb original in a presentation) | 23:28 |
| SpamapS | ok so I get all that | 23:28 |
| nibalizer | so I think we can land mtreinish's patches, service jenkins-subunit-worker-A stop, run puppet | 23:28 |
| *** salv-orlando has joined #openstack-infra | 23:28 | |
| nibalizer | oh hey infra-root I don't actually know the correct procedure for running puppet manually on a box, do we do that through ansible from the puppetmaster or just puppet agent --test ? | 23:29 |
| SpamapS | but now I'm remembering that I want to write a couple of entries into graphite... | 23:29 |
| *** hichihara has joined #openstack-infra | 23:29 | |
| nibalizer | mtreinish: do you and james turnbull get confused ever? kartar vs kortar ? | 23:29 |
| SpamapS | and this isn't an example of something that writes to graphite | 23:29 |
| jeblair | nibalizer: you can still use agent --test, at least until we remove the master | 23:30 |
| nibalizer | jeblair: sweet thanks | 23:30 |
| clarkb | nibalizer: `puppet agent --test` | 23:30 |
| nibalizer | that is something we sudo -i for right? because something something puppet sucks? | 23:30 |
| mtreinish | SpamapS: super hand wavy you replace the subunit2sql bits (or the logstash bits) with the talk to graphite bits | 23:30 |
| clarkb | nibalizer: yes | 23:31 |
| *** salv-orlando has quit IRC | 23:31 | |
| mtreinish | nibalizer: puppet won't just do magic and take care of everything in 15min? | 23:31 |
| mtreinish | nibalizer: I don't think my stuff has ever been confused for his | 23:32 |
| mtreinish | don't know about vice versa though | 23:32 |
| *** erikwilson has quit IRC | 23:32 | |
| *** salv-orlando has joined #openstack-infra | 23:32 | |
| clarkb | nibalizer: some things like mysql don't handle having the $HOME being set wrong | 23:32 |
| SpamapS | and I can do that via the statsd protocol? | 23:33 |
| *** erikwilson has joined #openstack-infra | 23:33 | |
| *** salv-orlando has quit IRC | 23:34 | |
| mtreinish | SpamapS: dunno, I've never had to send graphite data before | 23:35 |
| SpamapS | seems like thats the way to do it | 23:36 |
| clarkb | ya you can statsd it | 23:36 |
| clarkb | zuul and ndepool have lots of example code | 23:36 |
| SpamapS | how are the current run stats submitted? | 23:36 |
| *** rfolco has joined #openstack-infra | 23:36 | |
| dolphm | i get this a lot when working with backports -- is that just because gerrit's (or git review's?) don't include the original branch name in the review branch/topic names? "Branch review/openstack_proposal_bot/openstack/requirements incorrectly tracking gerrit/stable/juno instead of gerrit/stable/kilo" | 23:36 |
| SpamapS | oh | 23:37 |
| SpamapS | there don't seem to be any stats in graphite for runs | 23:37 |
| clarkb | dolphm: when do you get that error? | 23:37 |
| clarkb | dolphm: what is the set of commands that ends up there? | 23:38 |
| dolphm | clarkb: in this case, it's doing `git review -d 225373`, i'm trying to find the review i checked out previously that leads to this.... | 23:38 |
| SpamapS | oh hm maybe zuul has per job checks | 23:38 |
| nibalizer | mtreinish: so will puppet restart the jenkins-subunit-worker? I don't think so | 23:39 |
| *** yamamoto_ has quit IRC | 23:39 | |
| nibalizer | according to http://git.openstack.org/cgit/openstack-infra/puppet-subunit2sql/tree/manifests/worker.pp#n70 it will restart if the subunit package gets updated or the config file changes | 23:39 |
| nibalizer | but not, you know, the script itself | 23:39 |
| mtreinish | SpamapS: there isn't any per individual run data in graphite, it's all aggregated at higher level. Like per job type, etc | 23:39 |
| SpamapS | right, I see that. | 23:39 |
| clarkb | dolphm: oh! I think it is because you have previously git review -d'd changes from other brnaches | 23:39 |
| *** dizquierdo has quit IRC | 23:40 | |
| clarkb | dolphm: so I think that is normal since we use a common topic. A git-review improvement would be to add the branch name to the branch git review uses on -d | 23:40 |
| mtreinish | nibalizer: hmm, ok. I just assumed it would I guess we can fix that, but there might be a reason we did that originally | 23:40 |
| * mtreinish doesn't remember | 23:40 | |
| clarkb | dolphm: so something like review/openstack_proposal_bot/openstack-requirements/stable/kilo | 23:40 |
| dolphm | clarkb: this will reproduce: $ git clone git@github.com:openstack/keystone.git ; cd keystone ; git review -d 227798 ; git review -d 225373 | 23:41 |
| *** dims has quit IRC | 23:41 | |
| dolphm | clarkb: so this is a bug in git review? | 23:41 |
| *** dims has joined #openstack-infra | 23:42 | |
| clarkb | dolphm: yes I think so | 23:42 |
| *** skylerberg has quit IRC | 23:42 | |
| *** camunoz has quit IRC | 23:42 | |
| SpamapS | ok so I could drop a file, something like 'runstats' and then have the jenkins event handler thing push that into a gearman worker which would push to statsd? Is there any reason that the jenkins event handler wouldn't just push those numbers right into statsd? | 23:43 |
| *** r-daneel has quit IRC | 23:43 | |
| *** otter768 has quit IRC | 23:43 | |
| *** Swami has quit IRC | 23:43 | |
| *** otter768 has joined #openstack-infra | 23:44 | |
| *** yamamoto has joined #openstack-infra | 23:44 | |
| mtreinish | dolphm: I ran that command and it didn't error: http://paste.openstack.org/show/476204/ | 23:44 |
| *** rfolco has quit IRC | 23:44 | |
| *** mestery has joined #openstack-infra | 23:45 | |
| dolphm | mtreinish: i was thinking it was dependent on the version of git - but i tried upgrading, and no it still failed. i've tried both 2.1.4, and 2.6.1. what are you using? | 23:45 |
| dolphm | i'm also on git-review 1.25.0 | 23:45 |
| mtreinish | dolphm: 2.6.1 | 23:45 |
| dolphm | mtreinish: what about git-review --version? | 23:46 |
| mtreinish | git review is 1.24 | 23:46 |
| dolphm | mtreinish: if i downgrade to that version, it doesn't fail. so this is a new bug | 23:47 |
| *** rfolco has joined #openstack-infra | 23:47 | |
| *** wenlock has quit IRC | 23:52 | |
| *** wenlock1 has quit IRC | 23:52 | |
| *** achanda has quit IRC | 23:53 | |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config: (Re)add midonet tempest jobs https://review.openstack.org/230932 | 23:54 |
| dolphm | does anyone working on git-review actually care about the bugs in story board? or, does anyone still work on git review? | 23:55 |
| clarkb | dolphm: it still gets patches and reviews | 23:55 |
| dolphm | without bug tracking? | 23:55 |
| clarkb | dolphm: its just slow as it mostly works fine now and generaly doesn't need new features | 23:55 |
| clarkb | dolphm: no I think storyboard is used | 23:55 |
| dolphm | it looks dead | 23:56 |
| nibalizer | still need another +2 on https://review.openstack.org/#/c/234469/ btw | 23:58 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
