| *** dingyichen has joined #openstack-infra | 00:00 | |
| tonyb | sridhar_ram: I can push the chnage for you if you like. | 00:01 |
|---|---|---|
| pabelanger | fungi: ya, I seen it, but haven't ready up on the settings. I can do that in the morning and propose a follow up | 00:01 |
| *** zhurong has quit IRC | 00:01 | |
| *** zhurong has joined #openstack-infra | 00:02 | |
| sridhar_ram | tonyb: thanks for the offer, but i'm pushing it right now.. tox run in progress | 00:02 |
| sridhar_ram | tonyb: which, btw, taking a while looping thru' bunch of "Checking git repo .." | 00:02 |
| tonyb | sridhar_ram: okay, I'll look out for it. | 00:03 |
| sridhar_ram | tonyb: thanks! | 00:03 |
| tonyb | sridhar_ram: for the record I foudn that it shoudl be done near line 3038 | 00:03 |
| * sridhar_ram is checking.. | 00:04 | |
| tonyb | sridhar_ram: as there are a bunch of projects in your situation, so grouping them is good | 00:04 |
| tonyb | sridhar_ram: I only just found them whiel doign the research .... | 00:04 |
| sridhar_ram | tonyb: I realized i did something like this earlier.. my memory gave up on me :( | 00:04 |
| *** zhurong has quit IRC | 00:04 | |
| tonyb | :) | 00:04 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config: Update logrotate for mirror server https://review.openstack.org/357453 | 00:05 |
| pabelanger | fungi: gate failure^ | 00:05 |
| *** wcriswell has joined #openstack-infra | 00:05 | |
| sridhar_ram | tonyb: i ended up adding at L2633 alone w/ few other things | 00:05 |
| sridhar_ram | tonyb: what do you think ? | 00:06 |
| tonyb | sridhar_ram: I can see it going either way but I think keeping all the releasenotes exclusion jobs together is slightly better than keeping all the tacker jobs togther | 00:07 |
| sridhar_ram | tonyb: fair enough, will move it there | 00:07 |
| tonyb | sridhar_ram: but I'm just an interested bystander so could easily be wrong | 00:08 |
| *** itisha has joined #openstack-infra | 00:08 | |
| sridhar_ram | tonyb: no, that is actually a good find.. will put it next to manila-ui in http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/layout.yaml#n3051 | 00:10 |
| *** thorst_ has joined #openstack-infra | 00:10 | |
| tonyb | sridhar_ram: sure | 00:12 |
| openstackgerrit | Merged openstack-infra/puppet-mediawiki: apache: Don't just send 403s on Apache 2.4 https://review.openstack.org/352033 | 00:12 |
| Krenair | hurrah | 00:15 |
| *** rajinir has quit IRC | 00:15 | |
| ianw | clarkb / greghaynes: one quite odd thing in that strace -> 3879 open("/opt/dib_tmp/dib_build.y8yDOp9f/du_output.tmp.mb", O_RDONLY) = 75 <- that's a big number... | 00:15 |
| *** stewie925 has joined #openstack-infra | 00:15 | |
| *** skipp has joined #openstack-infra | 00:16 | |
| *** thorst__ has joined #openstack-infra | 00:17 | |
| openstackgerrit | Sridhar Ramaswamy proposed openstack-infra/project-config: Disable releasenotes jobs for tacker https://review.openstack.org/357550 | 00:17 |
| sridhar_ram | tonyb: ^^^ | 00:17 |
| tonyb | sridhar_ram: Thanks | 00:18 |
| openstackgerrit | Chris Krelle proposed openstack/diskimage-builder: WIP: A hardware burn-in element. https://review.openstack.org/355675 | 00:18 |
| *** thorst_ has quit IRC | 00:20 | |
| *** kzaitsev_mb has quit IRC | 00:21 | |
| *** _sarob has quit IRC | 00:22 | |
| *** yamahata has quit IRC | 00:22 | |
| *** thorst__ has quit IRC | 00:23 | |
| jogo | pbr question, I want to include ChangeLog in my descrpition-file, but when building the sdist pbr complains the file isn't there yet | 00:25 |
| jogo | how do I force it to generate the file first? | 00:25 |
| jogo | mordred: ^ | 00:26 |
| *** raunak has quit IRC | 00:29 | |
| mordred | zomg. it's a jogo | 00:29 |
| mordred | jogo: we had a patch to pbr a while ago to do that for you automatically | 00:30 |
| jogo | mordred: o/ I tried using pbr for something but its caused me issues | 00:30 |
| mordred | boo. I want it to not cause you issues | 00:30 |
| fungi | mordred: in a similar vein, where did we end up on support for getting autogenerated manpages included in sdists? | 00:30 |
| *** beagles_biaw is now known as beagles | 00:31 | |
| jogo | https://travis-ci.org/pinterest/git-stacktrace/builds/153440617 | 00:31 |
| *** yamahata has joined #openstack-infra | 00:31 | |
| mordred | jogo: https://review.openstack.org/#/c/95569/ | 00:31 |
| jogo | is the bug I am seeing | 00:31 |
| mordred | jogo: I can put that back onto the stack - I gave up because I did it right around when we started refactoring pbr | 00:32 |
| jogo | the error I am seeing is in pbr/utils | 00:32 |
| *** jamielennox is now known as jamielennox|away | 00:33 | |
| fungi | yeah, i have some non-openstack python projects i maintain where i'd like to use pbr, but would want autogenerated changelogs and manpages in my sdists. maybe i'll find time to review or hack on these features soon | 00:33 |
| jogo | in https://github.com/openstack-dev/pbr/blob/master/pbr/util.py#L279 | 00:33 |
| mordred | oh - also - you seem to be using an old pbr | 00:34 |
| jogo | mordred: also while adding version automatically is sorta neat, I was just going to do https://github.com/openstack-dev/pbr/blob/64699d79bec5f0955566aa9129a6b9dbf686035f/pbr/tests/testpackage/setup.cfg | 00:34 |
| fungi | indeed... 1.10 | 00:34 |
| fungi | er | 00:35 |
| fungi | 1.10.0 is current | 00:35 |
| jogo | https://pypi.python.org/pypi/pbr | 00:35 |
| fungi | according to https://pypi.org/project/pbr/ | 00:35 |
| *** zz_dimtruck is now known as dimtruck | 00:35 | |
| fungi | it's 0.10 that was ancient | 00:35 |
| mordred | yah - but ... | 00:36 |
| mordred | pbr (0.5.2.5.g5b3e942)' is being parsed as a legacy, non PEP 440, version. | 00:36 |
| jogo | yeah I am using the latest (just confirmed) | 00:36 |
| mordred | I do not expect to see that warning - but that's not the issue | 00:36 |
| mordred | jogo: yah - to do the thing you want, we need to finish the patch I linked, or right a new one that groks that you want to do the changelong into description-file so generates it first | 00:36 |
| jogo | mordred: I would prefer the latter | 00:37 |
| mordred | it shouldn't be super-hard to special case that | 00:37 |
| mordred | yah | 00:37 |
| jogo | as I don't want to limit things to a arbitrary number of versions mordred things is enough :) | 00:37 |
| fungi | maybe that version number is not coming from pbr? i thought it would only handle 3-component semver version numbers plus dev extensions anyway | 00:37 |
| mordred | yah. I'm not sure what it's coming from | 00:37 |
| jogo | shouldn't the change just be: generate files before creating sdist? | 00:38 |
| *** Hal has quit IRC | 00:38 | |
| *** Hal has joined #openstack-infra | 00:38 | |
| mordred | jogo: oh - see, you're talking like the systems involved have elements of sanity | 00:39 |
| fungi | oh, wait, it actually says it's having trouble parsing "pbr (0.5.2.5.g5b3e942)" | 00:39 |
| fungi | so maybe there's a very old pbr preinstalled? | 00:39 |
| mordred | yah. that was my concern | 00:39 |
| openstackgerrit | Merged openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/357032 | 00:39 |
| mordred | I know basically nothing about travis, so I'm not a huge help in interpreting that | 00:39 |
| openstackgerrit | Changcheng Intel proposed openstack-infra/jenkins-job-builder: add post-send script option https://review.openstack.org/355135 | 00:39 |
| jogo | mordred: old version may be an issue, but I can repro with the latest | 00:39 |
| jogo | locally | 00:40 |
| fungi | and then it gets replaced by pbr-1.10.0-py2.7.egg | 00:40 |
| openstackgerrit | Merged openstack-infra/project-config: Nodejs6 jobs are now voting. https://review.openstack.org/356616 | 00:40 |
| openstackgerrit | Merged openstack-infra/project-config: Remove python3 jobs for zuul https://review.openstack.org/355239 | 00:40 |
| openstackgerrit | Merged openstack-infra/project-config: Remove python3 jobs from nodepool https://review.openstack.org/355449 | 00:40 |
| mordred | yah. sorry - the two aren't related - mostly just fixated on the bad text in the build output | 00:40 |
| mordred | :) | 00:41 |
| fungi | we never had a tag for 0.5.2 as far as i can tell | 00:41 |
| fungi | jumped straight from 0.5.0 to 0.5.6 for some reason, according to git | 00:41 |
| mordred | jogo: btw - you have skipsdist = False and usedevelop = True ... which is a little odd - it'll mean you'll have to deal with non-venv pbr for the sdist step, then venv pbr after that. | 00:42 |
| mordred | it won't fix this, but you may want to change that | 00:42 |
| mordred | jogo: so - unfortunately the description-file magic is happening really early | 00:42 |
| mordred | jogo: I agree, I think it shold be moved to later | 00:43 |
| *** pt_15 has joined #openstack-infra | 00:43 | |
| mordred | jogo: I'm not sure what the effect of changing the long_description value after the setup constructor has been called is | 00:43 |
| mordred | jogo: I _think_ it'll just be a class attribute and changeable | 00:43 |
| mordred | so it should be possible to do the easy version of what you're talking about without too much pain | 00:44 |
| mordred | but a little investigation will be needed | 00:44 |
| jogo | mordred: thanks for the sdist note | 00:44 |
| jogo | mordred: awesome, if you need any help testing this out etc, let me know what I can do to help | 00:45 |
| jogo | I naively figured because you had an example where you listed a changes file, this would just work ... | 00:46 |
| mordred | jogo: will do - it'll likely take me a few days to get to it fully - but I think the approach we talked about is a solid one and achievable | 00:46 |
| mordred | heh | 00:46 |
| *** jamielennox|away is now known as jamielennox | 00:46 | |
| *** nwkarsten has joined #openstack-infra | 00:48 | |
| *** gyee has quit IRC | 00:48 | |
| jogo | cool, I'll ping you tomorrow and see where things stand and how I can help | 00:50 |
| jogo | mordred: thanks! | 00:50 |
| mordred | \o/ | 00:52 |
| *** nwkarsten has quit IRC | 00:53 | |
| *** rbuzatu has joined #openstack-infra | 00:53 | |
| *** tqtran has quit IRC | 00:54 | |
| *** furlongm has quit IRC | 00:55 | |
| *** furlongm has joined #openstack-infra | 00:55 | |
| *** gouthamr has quit IRC | 00:57 | |
| *** rbuzatu has quit IRC | 00:57 | |
| tonyb | fungi: are the results of the infra mascot poll available somewhere? | 01:02 |
| mordred | tonyb: http://civs.cs.cornell.edu/cgi-bin/results.pl?id=E_c2c11d642eafb0e0 | 01:03 |
| tonyb | mordred: thank you kind sir | 01:03 |
| mordred | tonyb: fwiw, I think clearly we all meant to vote for "pink fairy armadillo" but something unexplained happened | 01:03 |
| *** caowei has joined #openstack-infra | 01:04 | |
| tonyb | mordred: :) | 01:04 |
| openstackgerrit | Merged openstack-infra/system-config: Update logrotate for mirror server https://review.openstack.org/357453 | 01:04 |
| *** tphummel has quit IRC | 01:06 | |
| fungi | yeah, i wasn't sensing a ton of enthusiasm, even though we had something like 25% voter turnout across ~800 people i think | 01:07 |
| fungi | and somehow an ant came out way ahead of all sorts of much cooler options | 01:07 |
| fungi | condorcet, how you have failed us | 01:08 |
| mordred | or | 01:08 |
| mordred | it has served us in ways we can't understand because math | 01:08 |
| * fungi puts blind faith in the math of the universe and goes on his way | 01:08 | |
| mordred | its job isn't to pick the best answer, it's job is to pick the least-hated answer :) | 01:08 |
| tonyb | :) | 01:08 |
| * mordred glares at queens | 01:08 | |
| mordred | I mean, we all know that quokka was a better choice there | 01:09 |
| mordred | but I guess math knows better | 01:09 |
| fungi | at least it wasn't quibboleth or something | 01:09 |
| *** zhurong has joined #openstack-infra | 01:09 | |
| mordred | fungi: I have just been informed, fwiw, that Gluster's mascot is the ant | 01:09 |
| fungi | marvellous | 01:10 |
| fungi | one thing about ants... there are a lot of them to go around | 01:10 |
| mordred | fungi: https://twitter.com/gluster | 01:10 |
| fungi | maybe ours is the velvet ant https://en.wikipedia.org/wiki/Mutillidae | 01:11 |
| *** Apoorva_ has quit IRC | 01:12 | |
| mordred | fungi: I understand almost none of the summary paragraph of that article | 01:13 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config: Add source to ubuntu mirror distributions https://review.openstack.org/357560 | 01:13 |
| fungi | entomology is fin | 01:13 |
| fungi | fun too | 01:13 |
| fungi | also fine | 01:13 |
| fungi | on the other hand, ichthyology truly is fin | 01:14 |
| *** baoli has joined #openstack-infra | 01:14 | |
| *** salv-orlando has joined #openstack-infra | 01:15 | |
| *** salv-orlando has quit IRC | 01:15 | |
| *** salv-orlando has joined #openstack-infra | 01:16 | |
| *** pvaneck has quit IRC | 01:16 | |
| *** Apoorva has joined #openstack-infra | 01:17 | |
| *** salv-orl_ has quit IRC | 01:17 | |
| *** tbarron|afk is now known as tbarron | 01:18 | |
| *** spzala has joined #openstack-infra | 01:18 | |
| *** salv-orlando has quit IRC | 01:21 | |
| Krenair | fungi, did you make wiki-dev? | 01:22 |
| *** dimtruck is now known as zz_dimtruck | 01:22 | |
| fungi | Krenair: not yet, i'm going to boot it with the currently approved changes once they merge, then propose fixes for anything i find missing | 01:23 |
| Krenair | are you planning to clone wiki? or start afresh? | 01:23 |
| *** spzala has quit IRC | 01:23 | |
| fungi | i did at least get through reviewing all of them tonight, and they look great to me. thanks for writing those! | 01:23 |
| fungi | start afresh | 01:23 |
| fifieldt | fungi, thanks for the add :) | 01:23 |
| Krenair | okay | 01:23 |
| Krenair | so you'll have to figure out the diffs for yourself, no puppet help :/ | 01:24 |
| fifieldt | unfortuntely cannot use as https://wiki.openstack.org/wiki/Special:UserRights is blank screening ( 500ing I think) on POST :( | 01:24 |
| fungi | i'll boot a basic ubuntu trusty instance using our launch script, and name it wiki.openstack.org so it'll get the puppet module applied | 01:24 |
| *** chlong has joined #openstack-infra | 01:24 | |
| fungi | fifieldt: i saw your comment from yesterday while i was asleep. thanks for the reminder, taking a look now | 01:24 |
| fifieldt | oh, sorry to appear pushy :( | 01:25 |
| fungi | fifieldt: not pushy at all | 01:25 |
| fungi | after login, when i go to https://wiki.openstack.org/wiki/Special:UserRights i see a form to enter a username. entering my username pulls up the expected permissions list. what are you getting? | 01:26 |
| fungi | fifieldt: ^ | 01:26 |
| fifieldt | display works | 01:26 |
| fifieldt | POST the form | 01:26 |
| fifieldt | with a new box ticked | 01:26 |
| fifieldt | and the reason filled in | 01:26 |
| fungi | oh, like to alter one of the permissions there? | 01:26 |
| fifieldt | ya | 01:26 |
| *** nwkarsten has joined #openstack-infra | 01:26 | |
| fifieldt | eg add autopatrol to https://wiki.openstack.org/wiki/Special:UserRights/Alan_Clark | 01:27 |
| fungi | "The user rights of Fungi have been saved. | 01:27 |
| fungi | " | 01:27 |
| fungi | i added autopatrol for my account and added a reason | 01:27 |
| fifieldt | I've tried with 3 different users and got the same result :( | 01:27 |
| fungi | https://wiki.openstack.org/w/index.php?title=Special%3AUserRights&user=fungi shows it in the logs for me | 01:27 |
| fungi | fifieldt: what's your wiki username? | 01:27 |
| fifieldt | probably https://wiki.openstack.org/wiki/Special:UserRights/fifieldt | 01:28 |
| fungi | ahh, it's fifieldt | 01:28 |
| fungi | indeed | 01:28 |
| Krenair | fungi, there'll be an error going into the logs when fifieldt gets the HTTP 500. Can you dump it here or PM it to me? | 01:28 |
| fungi | i should have just tried it ;) | 01:28 |
| Krenair | apache error log | 01:28 |
| fifieldt | :P | 01:28 |
| fungi | indeed, when i try to add you to the autopatrol group, i too get a blank page when submitting | 01:29 |
| fungi | Krenair: digging it up now | 01:29 |
| fungi | not in /var/log/apache2/mediawiki-error.log at least | 01:29 |
| fungi | here we go, it's in the general apache error.log | 01:30 |
| Krenair | this is an unpuppetised machine... does the apache config specify actually using mediawiki-error.log? | 01:31 |
| fungi | this showed up when i tried to set fifieldt to be an autopatrol group member: http://paste.openstack.org/show/560957/ | 01:31 |
| fifieldt | :| | 01:32 |
| *** Apoorva has quit IRC | 01:32 | |
| fungi | Krenair: yeah, apache vhost config has ErrorLog /var/log/apache2/mediawiki-error.log (which i believe is actually from our puppet module) | 01:32 |
| Krenair | yes | 01:32 |
| fungi | Krenair: some errors are landing in there, but this particular error seems to have gone to the general apache error.log instead | 01:33 |
| Krenair | is the distinction exceptions vs. fatals? | 01:33 |
| fifieldt | line 117 of that file appears to be trying to send an email | 01:33 |
| fifieldt | https://doc.wikimedia.org/mediawiki-core/1.26.0/php/UserMailer_8php_source.html | 01:33 |
| Krenair | yeah it may be the code that notifies the user of a change to them | 01:34 |
| fifieldt | "This module performs a direct (authenticated) login to a SMTP Server to use for mail relaying if $wgSMTP specifies an array of parameters. It requires PEAR:Mail to do that. Otherwise it just uses the standard PHP 'mail' function." | 01:34 |
| fungi | Krenair: maybe, yeah. the ones showing up in mediawiki-error.log seem to be things like "AH00126: Invalid URI in request [...]" and "AH01630: client denied by server configuration [...]" | 01:34 |
| Krenair | I think those are actually apache errors | 01:35 |
| fungi | they look like it | 01:35 |
| Krenair | Ohh | 01:35 |
| fungi | so i guess errors coming from php are landing in the general error log and apache-level errors are going into the log for the vhost | 01:35 |
| Krenair | You do have the Echo extension installed.. | 01:36 |
| Krenair | And it's an old version | 01:36 |
| Krenair | Yeah okay | 01:36 |
| Krenair | You can't use Echo wmf/1.25wmf4 with core REL1_27 | 01:36 |
| fungi | i honestly have no idea why half the extensions on here were installed. Ryan_Lane set us up with a grab bag of extensions he thought we might use | 01:36 |
| Krenair | core was updated but not extensions | 01:36 |
| fungi | cool, i can find and replace that with a newer version | 01:37 |
| fungi | once we've got extensions coming from git per the puppet module this will presumably no longer be a concern anyway | 01:37 |
| * mordred is so excited for the new future wiki state | 01:38 | |
| Krenair | Well | 01:38 |
| Krenair | git won't automatically pull everything up to date all the time, though it could | 01:39 |
| fungi | grabbing https://extdist.wmflabs.org/dist/extensions/Echo-REL1_27-b87fa2f.tar.gz now | 01:39 |
| fungi | Krenair: thanks, that seems to have fixed it! | 01:41 |
| fungi | fifieldt: ^ | 01:41 |
| fifieldt | checking! | 01:41 |
| *** vinaypotluri has quit IRC | 01:41 | |
| fungi | i was able to add fifieldt to the autopatrol group anyway | 01:42 |
| Krenair | yay | 01:42 |
| fungi | i guess echo was breaking on certain accounts, or maybe just any account which wasn't my own | 01:42 |
| fifieldt | works perfectly | 01:42 |
| Krenair | It'd be accounts set up to get notifications of these things. | 01:42 |
| *** mtanino has quit IRC | 01:42 | |
| fifieldt | thank you very much! | 01:42 |
| fungi | Krenair: aha, makes sense | 01:42 |
| Krenair | It's configured by user preferences | 01:42 |
| fungi | i think i tuned down my notifications due to already getting thuosands of automated messages a day from so many of our systems | 01:42 |
| fungi | er, thousands that is | 01:43 |
| Krenair | your email address in preferences, and also notification preferences | 01:43 |
| Krenair | heh yeah I understand | 01:43 |
| Krenair | I'm not on wikimedia ops so I don't get root@ mail, but I still get a hell of a lot of mail from gerrit and phab | 01:43 |
| Krenair | I think they get a lot more | 01:44 |
| fungi | i do get notified about some wiki activities via e-mail, so my address is set up correctly, but i think i was very selective about what i had it notifying me of | 01:44 |
| *** shashank_hegde has quit IRC | 01:45 | |
| fungi | hrm, i have notifications set to let me know about user rights changes via e-mail, and the address there is correct and set to individual notifications | 01:45 |
| Krenair | it's at https://wiki.openstack.org/wiki/Special:Preferences#mw-prefsection-echo | 01:45 |
| Krenair | and you have "Individual notifications as they come in"? | 01:46 |
| fungi | that's what it's set to, yes | 01:46 |
| fungi | yep. i'm guessing it's just smart enough not to send me notifications of changes i make to myself or something | 01:46 |
| Krenair | That might be it | 01:47 |
| fungi | which would explain why i was able to alter my own group memberships, but as soon as i tried to alter someone else's it broke | 01:47 |
| Krenair | In fact... | 01:48 |
| *** raunak has joined #openstack-infra | 01:48 | |
| Krenair | ah, no, I didn't write that check in. but it does exist | 01:49 |
| fungi | heh | 01:49 |
| fungi | ugh, another puppet apply job on precise timed out and killed that whole patch series again | 01:50 |
| Krenair | don't you have a mechanism to merge without jenkins? | 01:51 |
| *** vinaypotluri has joined #openstack-infra | 01:51 | |
| fungi | and in osic again, but no v6 privacy address this time | 01:51 |
| fungi | Krenair: we do, but we prefer not to bypass our ci | 01:51 |
| fungi | since only our team can do that, and i wouldn't want the rest of the community to think we were giving our changes preferential treatment | 01:52 |
| Krenair | ah | 01:52 |
| fungi | and also, obviously, we want to take these as opportunities to diagnose what's breaking | 01:52 |
| fungi | this one ran in osic again, but no v6 privacy address this time so presumably got a new image. double-checking that assertion now | 01:53 |
| *** JerryOpenix has joined #openstack-infra | 01:53 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Fix typo for reporters doc https://review.openstack.org/353858 | 01:53 |
| *** yamahata has quit IRC | 01:53 | |
| fungi | hrm, the traceroute to git.o.o in it still failed though | 01:53 |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Add missing docs for maven-targets https://review.openstack.org/353844 | 01:54 |
| mordred | fungi: it had the privacy change and still borked the traceroute? | 01:54 |
| fungi | mordred: i'm not sure yet. looking to see if it simply didn't display a second v6 global address in the diagnostic output and is still running on an older image or something | 01:55 |
| pabelanger | ubuntu-xenial DIB is just finishing up now | 01:55 |
| pabelanger | should be able to upload shortly | 01:55 |
| fungi | note also this is precise, which we hadn't dug into much since we're the only ones still testing on it | 01:55 |
| pabelanger | Oh, ya. We still need to upload ubuntu-precise | 01:55 |
| Krenair | you're preparing to deploy xenial servers? | 01:56 |
| *** thorst_ has joined #openstack-infra | 01:56 | |
| Krenair | Yeah... Does puppet-mediawiki still need to test precise? :) | 01:56 |
| openstackgerrit | zhangyanxian proposed openstack-dev/pbr: Change assertTrue(isinstance()) by optimal assert https://review.openstack.org/357569 | 01:56 |
| pabelanger | Krenair: no, just uploading new images for nodepool | 01:56 |
| fungi | we're just wrapping up moving production services from precise to trusty, and also just starting to deploy some new services on xenial | 01:56 |
| Krenair | ah | 01:57 |
| pabelanger | ubuntu-xenial uploading to osic-cloud1 | 01:57 |
| fungi | but yeah, we should probably also annotate the global site manifest to no longer consider precise for puppet-mediawiki | 01:57 |
| openstackgerrit | zhangyanxian proposed openstack-dev/pbr: Change assertTrue(isinstance()) by optimal assert https://review.openstack.org/357569 | 01:57 |
| fungi | i'll submit that real quick | 01:57 |
| pabelanger | ubuntu-precise DIB started, not sure I'll be awake to upload it | 01:58 |
| fungi | pabelanger: mordred: we have a new ubuntu-precise image in osic-cloud1 as of almost 5 hours ago | 01:58 |
| fungi | so presumably has privacy addresses disabled | 01:58 |
| pabelanger | checking logs | 01:58 |
| *** thorst_ has quit IRC | 01:59 | |
| openstackgerrit | Merged openstack-infra/puppet-mediawiki: Add 'autopatrol' group https://review.openstack.org/356313 | 02:00 |
| *** esberglu has joined #openstack-infra | 02:01 | |
| pabelanger | fungi: it doesn't look like the DIB finished properly, so we might not be running 99-disable-rfc3041 | 02:01 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Test wiki.o.o manifest on Ubuntu Trusty https://review.openstack.org/357572 | 02:02 |
| fungi | pabelanger: odd that we would upload an image if dib didn't finish properly | 02:02 |
| *** fguillot is now known as fguillot_afk | 02:02 | |
| *** fguillot_afk has quit IRC | 02:02 | |
| *** fguillot has joined #openstack-infra | 02:04 | |
| Krenair | what happened to the idea to use phabricator here? I know it was discussed a while back | 02:04 |
| fungi | i wonder if it would make sense to refactor our infra-puppet-apply-jobs template to omit precise, and then only add the precise apply job for changes to modules still being used on that platform (which should be vanishingly small by now) | 02:04 |
| *** stewie925_ has quit IRC | 02:04 | |
| fungi | Krenair: we were only discussing using maniphest and pholio, not the rest of phabricator | 02:04 |
| *** zz_dimtruck is now known as dimtruck | 02:04 | |
| *** yanyanhu has joined #openstack-infra | 02:04 | |
| *** jamielennox is now known as jamielennox|away | 02:04 | |
| Krenair | well okay, maniphest and pholio :) | 02:04 |
| fungi | but storyboard development picked back up so we dropped the idea of migrating from launchpad to maniphest | 02:04 |
| Krenair | ah | 02:04 |
| fungi | we're still going to run pholio for the ui/ux team | 02:05 |
| fungi | it's nearly in place now | 02:05 |
| pabelanger | reuploading ubuntu-xenail to osic-cloud1, it failed for some reason | 02:07 |
| *** armax has joined #openstack-infra | 02:10 | |
| Krenair | you were going to have separate phabricator instances for pholio and maniphest? | 02:14 |
| *** rbuzatu has joined #openstack-infra | 02:15 | |
| Krenair | or just one phabricator instances, with most user-accessible non-maniphest/pholio applications disabled? | 02:15 |
| *** zhurong_ has joined #openstack-infra | 02:19 | |
| pabelanger | cloudnull: fungi: clarkb: ubuntu-xenial upload | 02:19 |
| *** zhurong has quit IRC | 02:20 | |
| *** rbuzatu has quit IRC | 02:21 | |
| *** esberglu has quit IRC | 02:23 | |
| *** rfolco has quit IRC | 02:25 | |
| ianw | greghaynes / clarkb : ah hah - i can replicate this. something in the xenial build has set stdout to O_NONBLOCK. that works until you flood a bunch of stuff that overflows pipe buffers, e.g. the du dump. filter tools like cat have no idea about EAGAIN, and boom | 02:26 |
| mordred | Krenair: the second thing- one intance with most user-accessible non-maniphest/pholio applications disabled | 02:26 |
| *** jamielennox|away is now known as jamielennox | 02:26 | |
| mordred | Krenair: but when things went back in favor of storyboard, the pholio stuff was stull interesting | 02:27 |
| ianw | greghaynes / clarkb : the problem is i guess -- removing the du dump fixes it ... for now. but it's going to hit us again. maybe not today, maybe not tomorrow, but it's coming :) | 02:27 |
| mordred | ianw: wow. | 02:27 |
| mordred | well, how much do we need the du dump? | 02:27 |
| mordred | oh - gah | 02:28 |
| mordred | I understand your sentence now | 02:28 |
| mordred | and I agree with your concern | 02:28 |
| ianw | mordred: yeah, other than running the entire xenial build under "strace -f" and looking for the O_NONBLOCK set | 02:28 |
| ianw | i guess that might work ... | 02:28 |
| mordred | ianw: god that would be a crazypants logfile :) | 02:29 |
| mordred | ianw: does strace descend into subshells like dib uses? | 02:29 |
| *** chlong has quit IRC | 02:29 | |
| ianw | mordred: "-f" should follow everything, iirc there's an option to put into logfiles separated with pids | 02:29 |
| *** sc` has joined #openstack-infra | 02:29 | |
| openstackgerrit | Thanh Ha proposed openstack-infra/jenkins-job-builder: Disentangle YamlParser and ModuleRegistry classes https://review.openstack.org/319622 | 02:30 |
| mordred | ianw: I suppose one giant file would be fine since all we need to do when it's done is grep for O_NONBLOCK :) | 02:30 |
| *** asettle has joined #openstack-infra | 02:30 | |
| ianw | it might be easier to instrument the kernel and log pids that set fd's to nonblock | 02:30 |
| sc` | ipv6 addresses get served to clients that do not possess ipv6 connectivity on the zuul status page. is this a known thing? | 02:31 |
| openstackgerrit | Thanh Ha proposed openstack-infra/jenkins-job-builder: Cleanup various deprecation warnings. https://review.openstack.org/319624 | 02:31 |
| ianw | on nodepool ~nodepool/logtst/test.py will replicate it | 02:32 |
| mordred | sc`: yup | 02:32 |
| mordred | sc`: the nodes in question only have ipv6 addresses | 02:32 |
| sc` | :D | 02:32 |
| sc` | yay future internet | 02:32 |
| *** shashank_hegde has joined #openstack-infra | 02:32 | |
| pabelanger | cloudnull: tomorrow we can focus on the SSH timeouts we are seeing in nodepool: | 02:32 |
| pabelanger | 2016-08-19 02:32:17,793 ERROR nodepool.NodeLauncher: Timeout launching node id: 3656221 in provider: osic-cloud1 error: Timeout waiting for ssh access | 02:32 |
| mordred | sc`: there will be a websocket interface for in-browser logs in the future | 02:32 |
| mordred | sc`: which will help those of us without the ipv6s | 02:33 |
| sc` | it tempts me to dive down the rabbit hole of getting ipv6 working in some form | 02:34 |
| *** asettle has quit IRC | 02:35 | |
| openstackgerrit | Merged openstack-infra/puppet-mediawiki: Puppetise LocalSettings.php -> ../Settings.php symlink https://review.openstack.org/352034 | 02:37 |
| *** elo has quit IRC | 02:38 | |
| pabelanger | So, it looks like I am not able to SSH into a new ubuntu-xenial image in osic-cloud1 | 02:39 |
| *** adreznec has left #openstack-infra | 02:39 | |
| pabelanger | I wonder is something with our urandom systemd is wonky | 02:39 |
| fungi | sc`: as someone whose isp has no v6, i'm personally quite satisfied with tunnelbroker.net (free encapsulated gre tunnel to hurricane electric) | 02:40 |
| cloudnull | evenings | 02:41 |
| pabelanger | just the person to ask | 02:41 |
| cloudnull | hows it ? | 02:41 |
| * cloudnull reading back | 02:42 | |
| pabelanger | cloudnull: just launched an ubuntu-xenial in osic-cloud1, but failing to SSH into it. You able to poke around on e5ccaaba-055c-4d55-88aa-7bebda8a8b5b ? | 02:42 |
| openstackgerrit | Thanh Ha proposed openstack-infra/jenkins-job-builder: Remove unused builder.Builder.update_job method https://review.openstack.org/319752 | 02:42 |
| openstackgerrit | Thanh Ha proposed openstack-infra/jenkins-job-builder: Rename Builder.delete_job to Builder.delete_jobs. https://review.openstack.org/319753 | 02:42 |
| openstackgerrit | Thanh Ha proposed openstack-infra/jenkins-job-builder: Merge builder.Builder and builder.Jenkins https://review.openstack.org/319754 | 02:42 |
| openstackgerrit | Thanh Ha proposed openstack-infra/jenkins-job-builder: Factor XmlJobGenerator out of YamlParser. https://review.openstack.org/319623 | 02:42 |
| openstackgerrit | Thanh Ha proposed openstack-infra/jenkins-job-builder: Disentangle YamlParser and ModuleRegistry classes https://review.openstack.org/319622 | 02:42 |
| openstackgerrit | Thanh Ha proposed openstack-infra/jenkins-job-builder: Be explicit about objects exported by modules. https://review.openstack.org/319625 | 02:42 |
| openstackgerrit | Thanh Ha proposed openstack-infra/jenkins-job-builder: Cleanup various deprecation warnings. https://review.openstack.org/319624 | 02:42 |
| openstackgerrit | Thanh Ha proposed openstack-infra/jenkins-job-builder: Make builder.Jenkins inherit from jenkins.Jenkins https://review.openstack.org/319751 | 02:42 |
| pabelanger | cloudnull: it is possible we incorrectly configured systemd | 02:42 |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/project-config: Add Neutron experimental job for scenario tests running on LinuxBridge https://review.openstack.org/357511 | 02:42 |
| cloudnull | pabelanger: sure . | 02:43 |
| cloudnull | let me get access to i t | 02:43 |
| * cloudnull grabs a beer | 02:43 | |
| *** elo has joined #openstack-infra | 02:43 | |
| fungi | i'm continually having to recheck infra puppet module changes for puppet apply jobs timing out on ubuntu-precise in osic-cloud1 | 02:44 |
| fungi | i have a feeling precise is behaving the same way trusty was | 02:44 |
| pabelanger | that is possible | 02:44 |
| cloudnull | I imagine that the privacy settings have been enabled within the kernel by default in ubuntu for a while . | 02:45 |
| fungi | yep, i suspect so | 02:45 |
| *** chlong has joined #openstack-infra | 02:46 | |
| openstackgerrit | Thanh Ha proposed openstack-infra/jenkins-job-builder: Rename "parallelize" decorator to "concurrent" https://review.openstack.org/333780 | 02:48 |
| cloudnull | fungi: pabelanger: are you seeing "ssh: connect to host $ADDRESS port 22: No route to host" ? | 02:48 |
| cloudnull | or something else? | 02:49 |
| pabelanger | ssh: connect to host 2001:4800:1ae1:18:f816:3eff:fec3:78a4 port 22: Connection timed out | 02:49 |
| pabelanger | that is using our newest ubuntu-xenail image | 02:49 |
| pabelanger | which includes urandom system change + ipv6 privacy | 02:50 |
| cloudnull | console log shows "[ 1.322642] systemd[1]: Configuration file /usr/lib/systemd/system/initialize-urandom.service is marked executable. Please remove executable permission bits. Proceeding anyway." | 02:50 |
| pabelanger | blarg | 02:50 |
| cloudnull | i wonder if systemd is not happy and failing to load other things? | 02:50 |
| pabelanger | I see that in DIB logs too | 02:50 |
| * cloudnull is by no means a systemd expert | 02:50 | |
| *** spzala has joined #openstack-infra | 02:51 | |
| * fungi blames lennart even harder | 02:51 | |
| pabelanger | cloudnull: you able to remove execute permissions, systemctl daemon-reload the reboot? | 02:51 |
| cloudnull | so that may be nothing more than a warning | 02:51 |
| cloudnull | I dont have keys on that VM however I can spawn a new image and bash it. | 02:51 |
| pabelanger | k | 02:51 |
| fungi | still worth correcting in our dib element | 02:51 |
| pabelanger | yup | 02:52 |
| pabelanger | doing that now | 02:52 |
| cloudnull | console log from the uuid of the instance http://cdn.pasteraw.com/flmdev66tgxfzrt99skkhw9lncx833u | 02:52 |
| pabelanger | I don't see sshd | 02:53 |
| pabelanger | or glean | 02:53 |
| fungi | in the short term, we should delete the images that won't boot so nodepool will fall back to its previous images | 02:54 |
| pabelanger | agreed | 02:54 |
| cloudnull | pabelanger: yea was just about to say that | 02:54 |
| cloudnull | RE: not seeing sshd | 02:55 |
| pabelanger | I also upload to ovh, I will delete them first | 02:55 |
| fungi | thanks! | 02:55 |
| cloudnull | a random sample of xenial images all show the similar boot logs | 02:55 |
| fungi | yeah, presumably the nova console log would show us | 02:56 |
| *** yuanying has quit IRC | 02:56 | |
| cloudnull | I'm just hitting nova console-log on various vms | 02:57 |
| fungi | `openstack server console-log <uuid>` or whatever it is | 02:57 |
| cloudnull | I still use `nova console-log $UUID` | 02:57 |
| cloudnull | I like the openstack client but old habits are hard to kill | 02:57 |
| *** elo has quit IRC | 02:57 | |
| fungi | i've been trying to train myself not to call the old clients | 02:58 |
| *** yuanying has joined #openstack-infra | 02:58 | |
| *** fguillot has quit IRC | 02:58 | |
| cloudnull | fungi: pabelanger: if you want access to an instance I use as a bastion I'd be happy to add keys. simple instnace w/ v4/6 public access which makes working on things easier. | 02:58 |
| * cloudnull still hates my ISP because they dont have v6 | 02:59 | |
| fungi | i have a very serviceable gre tunnel | 02:59 |
| fungi | to a v6 tunnel broker | 03:00 |
| cloudnull | huricane electric :) | 03:00 |
| pabelanger | cloudnull: can you confirm the permissiosn on /usr/local/bin/initialize-urandom.py ? | 03:00 |
| *** elo has joined #openstack-infra | 03:01 | |
| fungi | sounds like he doesn't have a means to do that since there's no console login access | 03:01 |
| pabelanger | Ah | 03:01 |
| fungi | but you should be able to mount the image on a loop dev on nodepool.o.o and inspect it | 03:02 |
| fungi | i think | 03:02 |
| pabelanger | ya | 03:02 |
| pabelanger | that is true | 03:02 |
| *** elo has quit IRC | 03:02 | |
| pabelanger | I think we didn't setup the right permissions on the python script | 03:03 |
| *** Sukhdev has quit IRC | 03:03 | |
| *** adreznec has joined #openstack-infra | 03:05 | |
| pabelanger | odd thing is, growroot.service works fine | 03:07 |
| pabelanger | and I used that as my template | 03:07 |
| openstackgerrit | Merged openstack-infra/project-config: Mark searchlight ES-1 test job non-voting https://review.openstack.org/356757 | 03:07 |
| cloudnull | so i can not access the vm either. | 03:07 |
| cloudnull | booted a couple instances using the image | 03:08 |
| cloudnull | nothing from the namespace | 03:09 |
| cloudnull | obviously nothing externally | 03:09 |
| cloudnull | console has no access, because no passwords are injected. | 03:10 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Fix file permissions with initialize-urandom element https://review.openstack.org/357583 | 03:10 |
| fungi | it's bedtime for me, so passing out in a bit, but i can probably pick this up after tomorrow's release team meeting if it's not been figured out | 03:10 |
| pabelanger | ianw: have a moment to look? ^ I am not sure why I need to chmod 0644 the systemd file, I used growroot as my template and it worked correctly | 03:11 |
| fungi | pabelanger: shouldn't it just be a matter of correcting the permissions on the file in the repo? | 03:11 |
| ianw | hmmm | 03:11 |
| pabelanger | fungi: that is the thing, they are 0644 | 03:11 |
| pabelanger | at least locally here | 03:11 |
| fungi | weird | 03:11 |
| ianw | it does seem like something the systemd install thing should deal with | 03:12 |
| *** _coolsvap_ has joined #openstack-infra | 03:13 | |
| *** _coolsvap_ is now known as coolsvap | 03:13 | |
| fungi | i wonder if someone reused some routine for installing initscripts (which do need to be executable) | 03:13 |
| pabelanger | okay, deleting uploading ubuntu-xenail images | 03:14 |
| *** elo has joined #openstack-infra | 03:14 | |
| ianw | pabelanger: oh, not using the systemd install stuff? | 03:15 |
| pabelanger | ianw: not sure I follow | 03:15 |
| *** ramishra has quit IRC | 03:16 | |
| ianw | pabelanger: oh, ok, you are. it seems like dib-init-system should do this? | 03:17 |
| pabelanger | ianw: I would expect it too, but http://nodepool.openstack.org/dib.ubuntu-xenial.log complains | 03:17 |
| *** raunak has quit IRC | 03:17 | |
| *** ramishra has joined #openstack-infra | 03:18 | |
| pabelanger | okay, broken xenial images removed | 03:18 |
| ianw | pabelanger: i mean it seems like a bug, but it does just do a cp? so should follow perms of file | 03:18 |
| cloudnull | ah. haha was just about to ask. | 03:19 |
| *** salv-orlando has joined #openstack-infra | 03:19 | |
| ianw | wait, it *has* extra bits? "Configuration file /usr/lib/systemd/system/initialize-urandom.service is marked executable. Please remove executable permission bits. Proceeding anyway." | 03:20 |
| *** elo has quit IRC | 03:20 | |
| ianw | old mode 100644 | 03:20 |
| ianw | new mode 100755 | 03:20 |
| *** ayoung has quit IRC | 03:22 | |
| *** yuanying has quit IRC | 03:22 | |
| ianw | pabelanger: dropped a comment, unless i'm reading it wrong, something doesn't add up... | 03:23 |
| *** raunak has joined #openstack-infra | 03:23 | |
| *** mriedem has quit IRC | 03:23 | |
| openstackgerrit | Merged openstack-infra/puppet-mediawiki: Also clone the vendor repository for newer MW versions https://review.openstack.org/352035 | 03:24 |
| *** elo has joined #openstack-infra | 03:25 | |
| pabelanger | ianw: we need 0755 on the python script, which I believe you think is the initialize-urandom.service file | 03:26 |
| *** salv-orlando has quit IRC | 03:27 | |
| *** raunak has quit IRC | 03:27 | |
| ianw | pabelanger: ok yeah ... but still, where is it coming from? | 03:29 |
| *** woodster_ has quit IRC | 03:29 | |
| pabelanger | I don't know | 03:29 |
| *** raunak has joined #openstack-infra | 03:29 | |
| pabelanger | The only thing I can think of, is initialize-urandom.py is missing execute permissions now | 03:30 |
| pabelanger | maybe systemd is confused | 03:30 |
| pabelanger | _shrugs_ | 03:30 |
| *** eandersson_ has joined #openstack-infra | 03:30 | |
| *** elo has quit IRC | 03:30 | |
| ianw | cp -RP $scripts_dir. $dest || true | 03:31 |
| *** honza|afk is now known as honza | 03:34 | |
| *** spzala has quit IRC | 03:34 | |
| ianw | ls -l /etc/nodepool/elements/initialize-urandom/init-scripts/systemd/initialize-urandom.service | 03:34 |
| ianw | -rwxr-xr-x 1 root root 285 Aug 18 20:28 /etc/nodepool/elements/initialize-urandom/init-scripts/systemd/initialize-urandom.service | 03:34 |
| *** baoli has quit IRC | 03:35 | |
| ianw | everything in /etc/nodepool/elements is 755??? | 03:36 |
| *** chem has quit IRC | 03:36 | |
| ianw | bad umask or something? | 03:36 |
| *** chem has joined #openstack-infra | 03:36 | |
| pabelanger | how is that possible | 03:36 |
| *** eandersson_ has quit IRC | 03:37 | |
| *** rbuzatu has joined #openstack-infra | 03:38 | |
| ianw | well i guess puppet populates it | 03:38 |
| openstackgerrit | Merged openstack-infra/puppet-mediawiki: Stick to REL1_27 on Trusty, not master https://review.openstack.org/352036 | 03:38 |
| pabelanger | grr | 03:39 |
| pabelanger | okay | 03:39 |
| pabelanger | that explains why growroot works properly | 03:39 |
| *** chlong has quit IRC | 03:39 | |
| pabelanger | yes | 03:40 |
| pabelanger | everything has 0755 on nodepool.o.o | 03:40 |
| pabelanger | https://github.com/openstack-infra/puppet-nodepool/blob/master/manifests/init.pp#L145 | 03:41 |
| pabelanger | that is why | 03:41 |
| pabelanger | so, we need to drop that | 03:41 |
| ianw | ahh, yes, recurse | 03:42 |
| ianw | ha ha, i hope we don't fix that thing then break something else | 03:42 |
| mordred | :) | 03:42 |
| mordred | I have no useful things to say | 03:42 |
| mordred | but I'm cheering you all on | 03:42 |
| *** rbuzatu has quit IRC | 03:42 | |
| pabelanger | well, my patch will 0644 the .service file now | 03:43 |
| pabelanger | so, we should be able to land that | 03:43 |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-nodepool: Stop managing permissions on /etc/nodepool/elements https://review.openstack.org/357588 | 03:45 |
| pabelanger | that will need some discussion | 03:45 |
| pabelanger | time to call it for the day | 03:46 |
| pabelanger | will pick this up in the morning | 03:46 |
| mordred | pabelanger: like, I don't like how recurse and 0755 are tied together | 03:48 |
| mordred | because what I _want_ that to say is that the directory is 0755 | 03:48 |
| ianw | pabelanger: cool ... progress. | 03:48 |
| mordred | not that the contents are | 03:48 |
| *** csomerville has joined #openstack-infra | 03:48 | |
| *** cody-somerville has quit IRC | 03:48 | |
| *** chlong has joined #openstack-infra | 03:52 | |
| *** yuanying has joined #openstack-infra | 03:54 | |
| ianw | mordred: yeah, sudo messes things up trying to trace the build | 03:59 |
| openstackgerrit | John L. Villalovos proposed openstack-infra/devstack-gate: Update local.conf for ironic-multinode case https://review.openstack.org/352790 | 03:59 |
| openstackgerrit | John L. Villalovos proposed openstack-infra/devstack-gate: DO NOT REVIEW https://review.openstack.org/356094 | 03:59 |
| ianw | systemtap seems like maybe the place to be | 04:00 |
| *** vinaypotluri has quit IRC | 04:01 | |
| *** armax has quit IRC | 04:06 | |
| *** armax has joined #openstack-infra | 04:07 | |
| *** armax has quit IRC | 04:07 | |
| *** armax has joined #openstack-infra | 04:07 | |
| *** armax has quit IRC | 04:08 | |
| *** chlong has quit IRC | 04:10 | |
| *** raunak has quit IRC | 04:12 | |
| *** TravT has quit IRC | 04:13 | |
| *** raunak has joined #openstack-infra | 04:13 | |
| *** links has joined #openstack-infra | 04:15 | |
| *** elo has joined #openstack-infra | 04:22 | |
| *** chlong has joined #openstack-infra | 04:23 | |
| *** javeriak has joined #openstack-infra | 04:25 | |
| *** javeriak has quit IRC | 04:29 | |
| *** salv-orlando has joined #openstack-infra | 04:32 | |
| *** dims has quit IRC | 04:38 | |
| *** salv-orlando has quit IRC | 04:39 | |
| *** _nadya_ has joined #openstack-infra | 04:43 | |
| *** dims has joined #openstack-infra | 04:43 | |
| *** kdas__ has joined #openstack-infra | 04:44 | |
| *** Sukhdev has joined #openstack-infra | 04:47 | |
| *** kdas__ is now known as kushal | 04:47 | |
| *** kushal has joined #openstack-infra | 04:47 | |
| openstackgerrit | Hieu LE proposed openstack-infra/project-config: Update Magnum jobs to include api-ref job https://review.openstack.org/357602 | 04:48 |
| *** coreyob has quit IRC | 04:51 | |
| *** armax has joined #openstack-infra | 04:52 | |
| *** bhunter71 has quit IRC | 04:52 | |
| *** coreyob has joined #openstack-infra | 04:54 | |
| *** armax has quit IRC | 04:58 | |
| *** armax has joined #openstack-infra | 05:01 | |
| *** hparekh has left #openstack-infra | 05:11 | |
| *** chlong has quit IRC | 05:12 | |
| *** ManishD has joined #openstack-infra | 05:22 | |
| *** sarob has joined #openstack-infra | 05:24 | |
| *** sdake_ has quit IRC | 05:24 | |
| openstackgerrit | Yanyan Hu proposed openstack-infra/project-config: Enable zaqar for senlin integration test https://review.openstack.org/354566 | 05:25 |
| *** chlong has joined #openstack-infra | 05:26 | |
| *** javeriak has joined #openstack-infra | 05:28 | |
| *** sarob has quit IRC | 05:28 | |
| *** armax has quit IRC | 05:36 | |
| *** salv-orlando has joined #openstack-infra | 05:38 | |
| tobiash | hi, is it possible somehow to set the OFFLINE_NODE_WHEN_COMPLETE flag depending on the node instead of the build job in zuul? | 05:41 |
| *** psachin has joined #openstack-infra | 05:43 | |
| *** rbuzatu has joined #openstack-infra | 05:44 | |
| *** r-mibu has quit IRC | 05:47 | |
| *** r-mibu has joined #openstack-infra | 05:47 | |
| *** rbuzatu has quit IRC | 05:49 | |
| *** rbuzatu has joined #openstack-infra | 05:52 | |
| *** chlong has quit IRC | 05:52 | |
| *** Ravikiran_K has joined #openstack-infra | 05:53 | |
| *** e0ne has joined #openstack-infra | 05:59 | |
| *** sdake has joined #openstack-infra | 05:59 | |
| *** nwkarsten has quit IRC | 06:00 | |
| *** nwkarsten has joined #openstack-infra | 06:01 | |
| *** _nadya_ has quit IRC | 06:01 | |
| *** YorikSar has quit IRC | 06:03 | |
| *** nwkarsten has quit IRC | 06:05 | |
| *** YorikSar has joined #openstack-infra | 06:05 | |
| *** DrifterZA has joined #openstack-infra | 06:05 | |
| *** e0ne has quit IRC | 06:07 | |
| *** e0ne has joined #openstack-infra | 06:07 | |
| *** javeriak has quit IRC | 06:08 | |
| *** sdake has quit IRC | 06:08 | |
| *** aeng has quit IRC | 06:12 | |
| *** javeriak has joined #openstack-infra | 06:14 | |
| *** e0ne_ has joined #openstack-infra | 06:14 | |
| *** javeriak has quit IRC | 06:15 | |
| *** e0ne has quit IRC | 06:15 | |
| *** valderrv has quit IRC | 06:15 | |
| *** caowei has quit IRC | 06:17 | |
| *** raunak has quit IRC | 06:17 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Move OSA os_nova func_lxd job to non-voting https://review.openstack.org/357625 | 06:20 |
| *** chlong has joined #openstack-infra | 06:22 | |
| *** apetrich has joined #openstack-infra | 06:22 | |
| openstackgerrit | Lukas Bednar proposed openstack-infra/jenkins-job-builder: Builders: Add ansible-playbook builder https://review.openstack.org/322243 | 06:23 |
| *** e0ne_ has quit IRC | 06:24 | |
| *** _nadya_ has joined #openstack-infra | 06:25 | |
| *** yamamoto has quit IRC | 06:31 | |
| *** adriant has quit IRC | 06:35 | |
| *** psachin has quit IRC | 06:35 | |
| *** pt_15 has quit IRC | 06:36 | |
| *** itisha has quit IRC | 06:40 | |
| *** rcernin has joined #openstack-infra | 06:41 | |
| cloudnull | im on my way to bed, but if someone from infra might be able to take a look, it seems the OSIC errornode launch attempts are quite high http://grafana.openstack.org/dashboard/db/nodepool-osic | 06:41 |
| *** changzhi has joined #openstack-infra | 06:41 | |
| cloudnull | maybe still related to the image rebuilding things for xenial that we're a little off earlier this evening ? | 06:42 |
| odyssey4me | jhesketh ping? ^ perhaps you can help? | 06:44 |
| openstackgerrit | Andrey Pavlov proposed openstack-infra/system-config: add openstack-ec2api irc channel to list of logged channels https://review.openstack.org/357636 | 06:45 |
| *** ccamacho has joined #openstack-infra | 06:46 | |
| yolanda | good moring | 06:46 |
| *** pilgrimstack has joined #openstack-infra | 06:54 | |
| *** javeriak has joined #openstack-infra | 06:55 | |
| *** e0ne has joined #openstack-infra | 06:55 | |
| *** jaosorior has joined #openstack-infra | 06:55 | |
| *** tesseract- has joined #openstack-infra | 06:56 | |
| *** yamahata has joined #openstack-infra | 07:00 | |
| odyssey4me | morning all | 07:00 |
| odyssey4me | yolanda when you have a minute, reviews of https://review.openstack.org/357283 & https://review.openstack.org/357625 would be appreciated | 07:01 |
| odyssey4me | simple job promotions | 07:01 |
| yolanda | hi odyssey4me . Going to start with the review queue in short | 07:02 |
| odyssey4me | thanks | 07:02 |
| *** _nadya_ has quit IRC | 07:02 | |
| jhesketh | odyssey4me, cloudnull: I'll take a look at osic | 07:04 |
| *** Sukhdev has quit IRC | 07:05 | |
| *** jeblair has quit IRC | 07:07 | |
| ManishD | HI Need help. I am getting comment from Elastic Recheck showing CI failure. Same CI was working properly mins before. Suddenly it's failing | 07:07 |
| ManishD | specifillcay gate-tempest-dsvm-postgres-full-ubuntu-xenial | 07:08 |
| *** esikachev has quit IRC | 07:08 | |
| *** pahuang has quit IRC | 07:14 | |
| *** jeblair has joined #openstack-infra | 07:15 | |
| *** salv-orl_ has joined #openstack-infra | 07:16 | |
| *** jaosorior has quit IRC | 07:17 | |
| *** kushal has quit IRC | 07:18 | |
| *** florianf has joined #openstack-infra | 07:18 | |
| *** salv-orlando has quit IRC | 07:19 | |
| *** javeriak has quit IRC | 07:20 | |
| *** salv-orl_ has quit IRC | 07:20 | |
| *** jaosorior has joined #openstack-infra | 07:22 | |
| *** jpich has joined #openstack-infra | 07:25 | |
| *** Na3iL has joined #openstack-infra | 07:29 | |
| *** javeriak has joined #openstack-infra | 07:30 | |
| openstackgerrit | Merged openstack-infra/project-config: Switch OSA Xenial jobs to voting https://review.openstack.org/357283 | 07:31 |
| openstackgerrit | Merged openstack-infra/project-config: Move OSA os_nova func_lxd job to non-voting https://review.openstack.org/357625 | 07:31 |
| openstackgerrit | Merged openstack-infra/system-config: Test wiki.o.o manifest on Ubuntu Trusty https://review.openstack.org/357572 | 07:34 |
| openstackgerrit | liyuanzhen proposed openstack-infra/jenkins-job-builder: Not limit stevedore to 1.8.0 https://review.openstack.org/357023 | 07:34 |
| openstackgerrit | Merged openstack-infra/project-config: Fix file permissions with initialize-urandom element https://review.openstack.org/357583 | 07:35 |
| openstackgerrit | Artur Zarzycki proposed openstack-infra/project-config: Add fuel-ccp-cinder repository https://review.openstack.org/357651 | 07:36 |
| sslypushenko | jhesketh: yolanda: Hi! I'm very asking you to review this patchset https://review.openstack.org/#/c/353861/. It is blocking issue for our team now... Can you take a look into it, please? Great thx! | 07:38 |
| openstackgerrit | liyuanzhen proposed openstack-infra/jenkins-job-builder: Not limit stevedore to 1.8.0 https://review.openstack.org/357023 | 07:38 |
| yolanda | sslypushenko, +2 | 07:39 |
| sslypushenko | yolanda: Thank you!! | 07:39 |
| jhesketh | sslypushenko: lgtm | 07:41 |
| jhesketh | sslypushenko: noting that it won't merge until the governance one does | 07:41 |
| jhesketh | err, sorry, other way around | 07:41 |
| openstackgerrit | Merged openstack-infra/project-config: Use python-db-jobs for networking-sfc https://review.openstack.org/354358 | 07:43 |
| sslypushenko | jhesketh Thank you so much! | 07:44 |
| *** dkehn has quit IRC | 07:44 | |
| *** pilgrimstack has quit IRC | 07:44 | |
| *** dkehn_ has quit IRC | 07:46 | |
| openstackgerrit | Merged openstack-infra/project-config: Temporary allow merges for sahara-tests-release https://review.openstack.org/357290 | 07:50 |
| openstackgerrit | Merged openstack-infra/project-config: Add repo for murano-pkg-check. Murano package validator tool. https://review.openstack.org/353861 | 07:50 |
| *** pilgrimstack has joined #openstack-infra | 07:50 | |
| *** dkehn has joined #openstack-infra | 07:51 | |
| *** dtantsur|afk is now known as dtantsur | 07:52 | |
| *** amotoki has joined #openstack-infra | 07:53 | |
| *** YorikSar has quit IRC | 07:53 | |
| openstackgerrit | Merged openstack-infra/project-config: Conditionally run gate-tempest-dsvm-neutron-linuxbridge in nova check https://review.openstack.org/354402 | 07:53 |
| *** ihrachys has joined #openstack-infra | 07:53 | |
| *** YorikSar has joined #openstack-infra | 07:54 | |
| *** shashank_hegde has quit IRC | 07:55 | |
| openstackgerrit | Merged openstack-infra/puppet-mediawiki: Puppetise extension repositories https://review.openstack.org/352055 | 07:55 |
| *** amotoki_ has joined #openstack-infra | 07:57 | |
| *** amotoki has quit IRC | 07:59 | |
| *** dkehn_ has joined #openstack-infra | 07:59 | |
| *** zzzeek has quit IRC | 08:00 | |
| *** zzzeek has joined #openstack-infra | 08:00 | |
| *** matthewbodkin has joined #openstack-infra | 08:00 | |
| *** matrohon has joined #openstack-infra | 08:01 | |
| *** yamahata has quit IRC | 08:03 | |
| *** openstackgerrit has quit IRC | 08:03 | |
| *** openstackgerrit has joined #openstack-infra | 08:03 | |
| openstackgerrit | Motohiro/Yuanying Otsuka proposed openstack-infra/project-config: Skip functional-magnum-k8s/swarm-ironic job https://review.openstack.org/357658 | 08:06 |
| openstackgerrit | Motohiro/Yuanying Otsuka proposed openstack-infra/project-config: Fix skipping files for magnum functional jobs https://review.openstack.org/357659 | 08:06 |
| *** cody-somerville has joined #openstack-infra | 08:07 | |
| *** cody-somerville has joined #openstack-infra | 08:07 | |
| *** csomerville has quit IRC | 08:10 | |
| *** changzhi has quit IRC | 08:10 | |
| *** mhickey has joined #openstack-infra | 08:14 | |
| *** e0ne has quit IRC | 08:14 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Set compute015.vanilla IP addresses https://review.openstack.org/357093 | 08:17 |
| openstackgerrit | Merged openstack-infra/system-config: Set compute012.vanilla IP addresses https://review.openstack.org/357088 | 08:17 |
| *** ggnel_t has quit IRC | 08:18 | |
| *** lucas-dinner is now known as lucasagomes | 08:18 | |
| *** armax has joined #openstack-infra | 08:22 | |
| openstackgerrit | Merged openstack-infra/system-config: Set compute007 IP addresses https://review.openstack.org/357075 | 08:22 |
| *** armax has quit IRC | 08:23 | |
| openstackgerrit | Motohiro/Yuanying Otsuka proposed openstack-infra/project-config: Fix skipping files for magnum functional jobs https://review.openstack.org/357659 | 08:24 |
| openstackgerrit | Motohiro/Yuanying Otsuka proposed openstack-infra/project-config: Skip functional-magnum-k8s/swarm-ironic job https://review.openstack.org/357658 | 08:24 |
| *** jordanP has joined #openstack-infra | 08:25 | |
| openstackgerrit | Merged openstack-infra/system-config: Set compute010.vanilla IP addresses and enabling it as it is functional https://review.openstack.org/357084 | 08:26 |
| *** Jeffrey4l has quit IRC | 08:27 | |
| *** markvoelker has quit IRC | 08:30 | |
| *** nmagnezi has joined #openstack-infra | 08:31 | |
| nmagnezi | hello | 08:31 |
| nmagnezi | is there an issue with some jobs? multiple jobs fail for my patch with the following http://logs.openstack.org/90/351490/11/check/gate-neutron-lbaasv2-dsvm-loadbalancer/7e0169c/console.html | 08:31 |
| *** Jeffrey4l has joined #openstack-infra | 08:32 | |
| *** Goneri has joined #openstack-infra | 08:33 | |
| *** derekh has joined #openstack-infra | 08:34 | |
| ManishD | @<nmagnezi>, same is the case with my patch | 08:36 |
| *** asettle has joined #openstack-infra | 08:36 | |
| ManishD | seems to be an infra issue | 08:36 |
| ManishD | I tried recheck, one or two CI processsed successful | 08:37 |
| nmagnezi | ManishD, i dud multiple rechecks and each time a different job fails with something similar to ^^ | 08:37 |
| ManishD | same here | 08:38 |
| *** eandersson_ has joined #openstack-infra | 08:41 | |
| *** Ravikiran_K has quit IRC | 08:43 | |
| *** DrifterZA has quit IRC | 08:43 | |
| *** DrifterZA has joined #openstack-infra | 08:44 | |
| *** hparekh has joined #openstack-infra | 08:44 | |
| *** hparekh_ has joined #openstack-infra | 08:44 | |
| *** Ravikiran_K has joined #openstack-infra | 08:45 | |
| *** hparekh has quit IRC | 08:45 | |
| *** hparekh_ has quit IRC | 08:45 | |
| *** jtomasek has joined #openstack-infra | 08:48 | |
| *** javeriak has quit IRC | 08:51 | |
| *** asettle has quit IRC | 08:53 | |
| *** asettle has joined #openstack-infra | 08:53 | |
| *** nmagnezi has quit IRC | 08:59 | |
| *** chlong has quit IRC | 09:03 | |
| *** electrofelix has joined #openstack-infra | 09:04 | |
| *** salv-orlando has joined #openstack-infra | 09:06 | |
| *** chem has quit IRC | 09:10 | |
| *** chem has joined #openstack-infra | 09:10 | |
| *** dimtruck is now known as zz_dimtruck | 09:14 | |
| *** eset has joined #openstack-infra | 09:18 | |
| eset | hello. People from openstack-ironic send me here to ask you .. I have problem with disk-image-create... getting /tmp/dib.build.xxxxx is busy and process stops | 09:20 |
| wznoinsk | eset: do you have more than one instance of dib running? could you share the log (paste.openstack.org ?) | 09:21 |
| eset | Ok I will check if there more instance | 09:21 |
| eset | and wait please for the log | 09:21 |
| *** kzaitsev_mb has joined #openstack-infra | 09:22 | |
| eset | wznoinsk: if there were any instance of dib I should see it in ps aux | 09:25 |
| eset | but there aren't any | 09:25 |
| *** sarob has joined #openstack-infra | 09:26 | |
| openstackgerrit | Merged openstack-infra/system-config: Set compute020.vanilla IP addresses https://review.openstack.org/357102 | 09:28 |
| openstackgerrit | Merged openstack-infra/puppet-infracloud: Search for VLAN devices on config drive to create bridge on it https://review.openstack.org/357339 | 09:29 |
| *** pilgrimstack has quit IRC | 09:29 | |
| *** sarob has quit IRC | 09:30 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Set compute018.vanilla IP addresses and enable it for deployment https://review.openstack.org/357098 | 09:31 |
| *** e0ne has joined #openstack-infra | 09:31 | |
| *** markvoelker has joined #openstack-infra | 09:31 | |
| eset | this my log http://paste.openstack.org/raw/561163/ | 09:31 |
| eset | It's the part of the process with moste important info where I get first Device or resourse busy | 09:32 |
| *** pilgrimstack has joined #openstack-infra | 09:33 | |
| *** ociuhandu has quit IRC | 09:33 | |
| *** pilgrimstack has quit IRC | 09:35 | |
| *** markvoelker has quit IRC | 09:36 | |
| wznoinsk | eset: have you tried to rm/unmount it manually after dib finishes? what 'lsof' tells you? | 09:38 |
| *** javeriak has joined #openstack-infra | 09:39 | |
| *** _nadya_ has joined #openstack-infra | 09:39 | |
| eset | I didn't unmount | 09:40 |
| *** changzhi has joined #openstack-infra | 09:41 | |
| *** pilgrimstack has joined #openstack-infra | 09:41 | |
| eset | lsof showed me only that two users from ssh where in tmp but not in tmp itself but deaper in difrent catalogs | 09:43 |
| *** ggnel_t has joined #openstack-infra | 09:43 | |
| eset | but I wonder why it can't access catalog which was generated during the disk-image-create process | 09:43 |
| *** jtomasek has quit IRC | 09:46 | |
| *** ianychoi has quit IRC | 09:47 | |
| *** jtomasek has joined #openstack-infra | 09:47 | |
| *** tosky has joined #openstack-infra | 09:48 | |
| *** javeriak has quit IRC | 09:53 | |
| *** jaosorior has quit IRC | 09:54 | |
| *** pilgrimstack has quit IRC | 09:55 | |
| johnthetubaguy | wondering if its time for gerrit to get restarted again | 09:55 |
| johnthetubaguy | it could be my dodgy wifi at the midcycle, but I got a 502 proxy error, slowness, but sometimes it works | 09:55 |
| *** pilgrimstack has joined #openstack-infra | 09:56 | |
| robcresswell | Came here to say the same thing ^^ | 09:57 |
| robcresswell | I'm just working from home and gerrit is really struggling. Multiple 502s. | 09:57 |
| wznoinsk | eset: do you have any errors in previous dib targets? (root.d/extra-data.d).. it may be some process hanging there still | 09:58 |
| tosky | also many jobs are failing right now with "[ERROR] /opt/stack/new/devstack/stackrc:784 Could not determine host ip address. See local.conf for suggestions on setting HOST_IP." | 09:58 |
| tosky | as some people already pointed out | 09:58 |
| *** M-docaedo_vector has quit IRC | 09:59 | |
| johnthetubaguy | robcresswell: at least its not just picking on me I guess :) | 10:00 |
| *** dtardivel has joined #openstack-infra | 10:03 | |
| *** changzhi has quit IRC | 10:03 | |
| rcarrillocruz | i'll try to restart it | 10:03 |
| robcresswell | johnthetubaguy: I've got service unavailable now :p | 10:03 |
| robcresswell | Ah, cool | 10:03 |
| rcarrillocruz | quite a bit of memory usage | 10:03 |
| rcarrillocruz | restarted it ok | 10:03 |
| rcarrillocruz | johnthetubaguy: robcresswell ^ | 10:04 |
| openstackgerrit | Swann Croiset proposed openstack-infra/project-config: Add push ACL for fuel-plugin-external-zabbix https://review.openstack.org/357690 | 10:04 |
| robcresswell | Thanks rcarrillocruz | 10:04 |
| johnthetubaguy | rcarrillocruz: thank you :) | 10:04 |
| rcarrillocruz | np | 10:04 |
| *** amoralej has joined #openstack-infra | 10:07 | |
| *** jtomasek has quit IRC | 10:08 | |
| eset | wznoinsk: I'm not having any problem in other server in deploying but it should work on of our node also. | 10:09 |
| eset | strange that none of the catalogs dib.builds are not removed after creating from /tmp | 10:10 |
| eset | they are still there | 10:11 |
| tosky | rcarrillocruz: was this gerrit issue related to the failure for (many? most?) devstack jobs right now? | 10:11 |
| rcarrillocruz | no, it shouldn't | 10:11 |
| *** yanyanhu has quit IRC | 10:11 | |
| tosky | uhm, and for the devstack issue, what could it be the reason? It's basically invalidating the reviews (and consuming resources) | 10:12 |
| *** jaosorior has joined #openstack-infra | 10:14 | |
| openstackgerrit | Aleksey Zvyagintsev proposed openstack-infra/project-config: ci-cd-pipeline-app-murano -remove noop jobs https://review.openstack.org/357740 | 10:14 |
| eset | All dib.build still are in /tmp | 10:15 |
| dougwig | HOST_IP failures appear to be on the osic cloud, from a random sample | 10:17 |
| *** sambetts|afk is now known as sambetts | 10:17 | |
| rcarrillocruz | http://logs.openstack.org/94/357094/1/check/gate-infra-puppet-apply-ubuntu-precise/ef0a551/console.html | 10:17 |
| rcarrillocruz | i'm also seeing similar issue to what we had some days ago on trusty/osic combination, but on precise | 10:17 |
| rcarrillocruz | cloudnull: ^ | 10:18 |
| *** pgadiya has joined #openstack-infra | 10:18 | |
| dougwig | can we disable the osic cloud while it's debugged? | 10:18 |
| wznoinsk | it looks like there should be at least two NIC in nodepool slaves, the failing ones have only one NIC hence devstack can't find HOST_IP properly | 10:19 |
| wznoinsk | rcarrillocruz: it looks like affecting osic nodes AFAICT | 10:19 |
| *** chem has quit IRC | 10:20 | |
| openstackgerrit | Merged openstack-infra/system-config: Set compute018.vanilla IP addresses and enable it for deployment https://review.openstack.org/357098 | 10:20 |
| *** sdague has joined #openstack-infra | 10:22 | |
| *** pilgrimstack has quit IRC | 10:23 | |
| *** chem has joined #openstack-infra | 10:24 | |
| openstackgerrit | Doug Wiegley proposed openstack-infra/project-config: Temporarily remove osic nodes for ubuntu-trusty|xenial https://review.openstack.org/357742 | 10:25 |
| dougwig | i'm not sure if enough infra cores are awake to comment, but ^^ :) | 10:26 |
| *** jlk has quit IRC | 10:26 | |
| *** M-docaedo_vector has joined #openstack-infra | 10:26 | |
| *** jlk has joined #openstack-infra | 10:26 | |
| *** jlk has quit IRC | 10:26 | |
| *** jlk has joined #openstack-infra | 10:26 | |
| amoralej | could it be related to the ip assigned to nics in the devstack instance, the ones failing have ip 10.1.14.* which are in the range of fixed_range | 10:26 |
| amoralej | FIXED_RANGE=10.1.0.0/20 | 10:26 |
| wznoinsk | btw. is there a working logstash/log search for gate? | 10:27 |
| rcarrillocruz | the problem is per flavor... | 10:29 |
| rcarrillocruz | http://logs.openstack.org/93/357093/2/gate/gate-infra-puppet-apply-fedora-23/19b79d4/console.html | 10:29 |
| rcarrillocruz | fedora goes thru | 10:29 |
| rcarrillocruz | i'm seeing the issue on precise | 10:29 |
| rcarrillocruz | can anyone link to trusty/xenial failures | 10:29 |
| rcarrillocruz | ? | 10:29 |
| *** psanchez has quit IRC | 10:29 | |
| rcarrillocruz | wznoinsk: http://logstash.openstack.org/ | 10:29 |
| dougwig | rcarrillocruz: i linked one in the review above. http://logs.openstack.org/69/319769/13/check/gate-tempest-dsvm-neutron-src-neutron-lib/4892a3b/ | 10:29 |
| rcarrillocruz | thanks | 10:30 |
| dougwig | that's trusty | 10:30 |
| wznoinsk | rcarrillocruz: http://logs.openstack.org/90/351490/11/check/gate-neutron-lbaasv2-dsvm-loadbalancer/7e0169c/logs/devstack-gate-setup-host.txt.gz | 10:31 |
| *** ihrachys has quit IRC | 10:31 | |
| openstackgerrit | Sergey Kraynev proposed openstack-infra/project-config: Add docker-suite-app-murano project to openstack https://review.openstack.org/357745 | 10:31 |
| *** psanchez has joined #openstack-infra | 10:31 | |
| amoralej | rcarrillocruz http://logs.openstack.org/93/357093/2/gate/gate-infra-puppet-apply-fedora-23/19b79d4/console.html is not running devstack, right? | 10:32 |
| *** markvoelker has joined #openstack-infra | 10:32 | |
| rcarrillocruz | ah good | 10:32 |
| rcarrillocruz | dougwig: i see you also remove precise on that change | 10:32 |
| rcarrillocruz | amoralej: yup, it's not just devstack | 10:32 |
| rcarrillocruz | osic ubuntus it seems | 10:32 |
| rcarrillocruz | yolanda , sdague : https://review.openstack.org/#/c/357742/1 | 10:33 |
| dougwig | rcarrillocruz: i was casting a wide net on the single node jobs, yes. | 10:33 |
| rcarrillocruz | amoralej: http://logs.openstack.org/93/357093/2/gate/gate-infra-puppet-apply-ubuntu-precise/a298de1/console.html , no devstack , ubuntu precise, osic, net failure | 10:35 |
| wznoinsk | rcarrillocruz: http://logstash.openstack.org/#dashboard/file/logstash.json?query=%5C%22Could%20not%20determine%20host%20ip%20address%5C%22 doesn't work for me, (/me feels stupid) | 10:35 |
| sdague | rcarrillocruz: hmmm... do you have an fail log I can look at? | 10:35 |
| rcarrillocruz | sdague: http://logs.openstack.org/94/357094/1/check/gate-infra-puppet-apply-ubuntu-precise/ef0a551/console.html exact issue as trusty the other day | 10:35 |
| rcarrillocruz | and others are reporting issues on ubuntu/osic (particularly dsvm) | 10:36 |
| rcarrillocruz | the one i pasted is something i hit myself | 10:36 |
| *** markvoelker has quit IRC | 10:36 | |
| amoralej | http://logs.openstack.org/13/357713/1/check/gate-aodh-dsvm-tempest-plugin-hbase/7a10879/console.html i.e. | 10:36 |
| dougwig | sdague: here's a trusty failure: http://logs.openstack.org/69/319769/13/check/gate-tempest-dsvm-neutron-src-neutron-lib/4892a3b/ | 10:36 |
| openstackgerrit | Merged openstack-infra/system-config: Set compute016.vanilla IP addresses https://review.openstack.org/357094 | 10:36 |
| amoralej | this is [ERROR] /opt/stack/new/devstack/stackrc:784 Could not determine host ip address. See local.conf for suggestions on setting HOST_IP. | 10:37 |
| sdague | ok - http://logs.openstack.org/69/319769/13/check/gate-tempest-dsvm-neutron-src-neutron-lib/4892a3b/console.html is trusty | 10:38 |
| *** zhurong_ has quit IRC | 10:38 | |
| sdague | rcarrillocruz: I see precise and trusty fails | 10:38 |
| sdague | but could not discover HOST_IP is different issue | 10:38 |
| rcarrillocruz | dougwig linked a xenial i think too | 10:38 |
| rcarrillocruz | not sure if related to osic or not | 10:38 |
| rcarrillocruz | dougwig: ? | 10:38 |
| dougwig | let me look for a xenial, we've had a bunch of failures here at the midcycle. only common thread i found was osic nodes. | 10:39 |
| amoralej | i think the dicover host_ip are only happening in osic | 10:39 |
| rcarrillocruz | thing is presumably the osic ipv6 issues were not present on xenial | 10:39 |
| amoralej | and may be related to wrong ips being assigned | 10:39 |
| amoralej | to VMs | 10:39 |
| rcarrillocruz | so not sure if those osic xenial issues are a different problem | 10:39 |
| sdague | rcarrillocruz: right, so there are 2 issues here, potentially | 10:40 |
| amoralej | assigned ips (as 10.1.14.211, 10.1.14.205, etc...) are in the range asigned as FIXED_RANGE | 10:40 |
| amoralej | that's an issue, i think | 10:40 |
| *** ihrachys has joined #openstack-infra | 10:40 | |
| sdague | right | 10:41 |
| sdague | well, the interface seems to have been given 10.0.0.0/8 for outbound | 10:41 |
| dougwig | sdague: rcarrillocruz: everything i'm finding is trusty osic. want me to narrow the patch? | 10:41 |
| rcarrillocruz | yeah | 10:42 |
| rcarrillocruz | cos xenial looks like a devstack problem | 10:42 |
| rcarrillocruz | whereas the trusty/precise are due to osic going ipv6 | 10:42 |
| rcarrillocruz | cloudnull was looking into it lately | 10:42 |
| sdague | rcarrillocruz: I was pretty sure he had a fix for that | 10:42 |
| rcarrillocruz | yeah, it merged | 10:42 |
| rcarrillocruz | but apparently the issue is still there | 10:42 |
| rcarrillocruz | and the precise image is quite recent | 10:42 |
| sdague | and it would have run on the precise image? | 10:43 |
| rcarrillocruz | let me get the exact numbers | 10:43 |
| sdague | because we were so focused on trusty | 10:43 |
| openstackgerrit | Doug Wiegley proposed openstack-infra/project-config: Temporarily remove osic nodes for ubuntu-trusty https://review.openstack.org/357742 | 10:43 |
| rcarrillocruz | http://paste.openstack.org/show/561238/ | 10:44 |
| rcarrillocruz | sdague: ^ | 10:44 |
| rcarrillocruz | i'm surprised as to why we are seeing on precise now | 10:44 |
| sdague | rcarrillocruz: well, maybe the fix breaks precise | 10:44 |
| rcarrillocruz | maybe it happened too earlier but because of having more trusty nodes we just focused on trusty, not sure | 10:44 |
| rcarrillocruz | sdague: that too | 10:45 |
| sdague | ok... so is there actually a trusty node that is showing that failure? | 10:45 |
| rcarrillocruz | oh no | 10:45 |
| sdague | because what I see is precise with the ipv6 node | 10:45 |
| rcarrillocruz | dougwig: we see precise and trusty | 10:45 |
| dougwig | rcarrillocruz: done | 10:45 |
| rcarrillocruz | so just remove xenial, sorry | 10:45 |
| dougwig | rcarrillocruz: alright, one sec. | 10:45 |
| sdague | and trusty / xenial | 10:45 |
| *** gildub has quit IRC | 10:45 | |
| *** gildub_ has quit IRC | 10:45 | |
| sdague | in osic doing different issues, which is the issue amoralej is pointing out | 10:46 |
| rcarrillocruz | ok | 10:46 |
| rcarrillocruz | dougwig: just precise | 10:46 |
| openstackgerrit | Doug Wiegley proposed openstack-infra/project-config: Temporarily remove osic nodes for ubuntu-trusty|precise https://review.openstack.org/357742 | 10:46 |
| rcarrillocruz | let's prep a patch for devstack trusty/xenial | 10:46 |
| rcarrillocruz | :D i'm giving you a hard time dougwig | 10:46 |
| sdague | which is that osic has started handing out in ranges that we're using for fixed ips | 10:46 |
| sdague | and... yes, if your ip address is in the fixed ips ranges, host_ip detection fails | 10:47 |
| rcarrillocruz | ++ | 10:47 |
| sdague | but we can fix that with a devstack gate change | 10:47 |
| rcarrillocruz | yeah | 10:47 |
| sdague | rcarrillocruz: do you know the fixed range that providers are using, so we can use something other than 10.1 here? | 10:48 |
| rcarrillocruz | i can hold a node and see | 10:48 |
| sdague | rcarrillocruz: no, we need the different provider settings | 10:48 |
| rcarrillocruz | that i'm not sure i'm afraid | 10:48 |
| rcarrillocruz | never checked that before | 10:48 |
| *** john-davidge has joined #openstack-infra | 10:49 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/project-config: Temporarily remove osic nodes for ubuntu-precise https://review.openstack.org/357742 | 10:51 |
| openstackgerrit | Sean Dague proposed openstack-infra/devstack-gate: Change fixed ip range https://review.openstack.org/357764 | 10:51 |
| rcarrillocruz | dougwig: i pushed a patchset to your change, let's just remove precise on osic | 10:51 |
| rcarrillocruz | the other issues will be handled by a devstack-gate change ^ | 10:51 |
| dougwig | rcarrillocruz: as long as we're pushing the d-g fix, which is better, that's cool. i just figured it was the middle of the night for folks. | 10:52 |
| rcarrillocruz | sdague: https://review.openstack.org/#/c/357742/4 | 10:52 |
| rcarrillocruz | pls | 10:52 |
| sdague | rcarrillocruz: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22%20%20proto%20kernel%20%20scope%20link%20%20src%20%5C%22 - it looks like 10.199.0.0/20 should be safe | 10:54 |
| rcarrillocruz | oh nice, good query | 10:54 |
| dougwig | ugh, d-g gate is much longer than p-c's. oh well. :) | 10:55 |
| sdague | dougwig: yeh, but dropping the osic gate capacity isn't going to be great either | 10:56 |
| sdague | rcarrillocruz: you able to jump the d-g change to the gate? | 10:56 |
| dougwig | sdague: well, it's effectively worse than dropped right now anyway. | 10:56 |
| rcarrillocruz | i sure can | 10:56 |
| dougwig | it does clear every job in 13 minutes, though. :) | 10:57 |
| *** yamamoto has joined #openstack-infra | 10:58 | |
| odyssey4me | rcarrillocruz dougwig IIRC I saw chatter over the evening into the morning about an image update that caused issues, then the new image had to be deleted because it caused issues | 10:58 |
| rcarrillocruz | sdague: just enqueued in gate | 10:58 |
| odyssey4me | it's possible that the deletion wasn't done everywhere | 10:58 |
| rcarrillocruz | odyssey4me: oh | 10:59 |
| odyssey4me | butyes, when I spoke to cloudnull this morning he noted that there was a dramatic increase in node launch failures | 10:59 |
| rcarrillocruz | ok | 10:59 |
| rcarrillocruz | let's wait for fungi and others | 10:59 |
| sdague | rcarrillocruz: don't you have to do a thing manually to jump it to gate before tests complete? | 11:00 |
| rcarrillocruz | i've run the zuul enqueue command, i was expecting that to move it to gate | 11:00 |
| rcarrillocruz | yolanda: ^ | 11:00 |
| rcarrillocruz | ? | 11:00 |
| rcarrillocruz | zuul enqueue --trigger gerrit --pipeline gate --project openstack-infra/devstack-gate --change 357764,1 | 11:01 |
| sdague | ok, idk | 11:01 |
| sdague | sometimes it takes a minute when things are slow | 11:01 |
| odyssey4me | I'm surprised that precise is still running jobs. | 11:02 |
| amrith | hi infra, would someone please put the second +2 on https://review.openstack.org/#/c/356407/. Yolanda put one on it yesterday, it depended on a change which has now merged. thx | 11:03 |
| dougwig | ok, notified folks here to stop with the recheck spam, which won't help things. | 11:03 |
| sdague | odyssey4me: there are still infra servers that run on it | 11:03 |
| yolanda | sorry, i was out. So rcarrillocruz is it working? | 11:03 |
| sdague | amrith: +A | 11:03 |
| rcarrillocruz | yolanda: it's taking time to be moved to gate | 11:04 |
| sdague | yolanda: it's taking too long honestly | 11:04 |
| rcarrillocruz | that enqueue command should suffice, no? | 11:04 |
| amrith | sdague, rcarrillocruz thank you | 11:04 |
| sdague | rcarrillocruz: oh, you didn't +A first | 11:04 |
| sdague | I think it will still count on that | 11:04 |
| sdague | rcarrillocruz: want to try again? | 11:05 |
| sdague | I just added the +A | 11:05 |
| rcarrillocruz | and i see this on the log | 11:05 |
| rcarrillocruz | 2016-08-19 10:57:50,373 INFO zuul.Scheduler: Adding openstack-infra/devstack-gate, <Change 0x7fc189727690 357764,1> to <Pipeline gate> | 11:05 |
| rcarrillocruz | i can sure | 11:05 |
| *** zz_dimtruck is now known as dimtruck | 11:05 | |
| sdague | right, but zuul still enforces the base logic | 11:05 |
| sdague | dougwig: so... I'm assuming this is because neutron ip allocation algorithm keeps moving through the empty space instead of reusing existing ips | 11:06 |
| rcarrillocruz | sdague: now is on gate | 11:07 |
| sdague | I guess we could calculate how long it would take osic to get to the other side of the problem range :) | 11:07 |
| rcarrillocruz | it was just slow | 11:07 |
| dougwig | sdague: it actually got worse lately from my perspective, and went totally random. i think they backed that madness out, though. | 11:07 |
| dougwig | sdague: but yes, it cycles the entire range before restarting. | 11:07 |
| sdague | well, using the empty space is actually useful, because we had issues on n-net where dnsmasq looses release events | 11:08 |
| sdague | so if you agressively reuse existing space, you end up with other issues | 11:08 |
| dougwig | neat, i've never seen a job in check and gate at the same time. | 11:08 |
| dougwig | sdague: if anything the ipam folks want to less aggressively reuse, because of the synchronization issues around finding the next open slot under high load. we had to go back to sequential because some other projects had hard-coded assumptions that we give out sequentially, and we wanted to give them time to stop doing that. | 11:11 |
| sdague | huh, interesting. What assumptions were being made on ip hand out? | 11:12 |
| *** pilgrimstack has joined #openstack-infra | 11:13 | |
| openstackgerrit | Merged openstack-infra/project-config: Update trove jobs to include api-ref job https://review.openstack.org/356407 | 11:13 |
| dougwig | sdague: tests were assuming sequential on new networks. heat also had a similar assumption somewhere. | 11:14 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: Use separated SSL endpoint environment file https://review.openstack.org/356488 | 11:15 |
| *** dimtruck is now known as zz_dimtruck | 11:15 | |
| yolanda | #status notice Precise tests on OSIC provider are currently failing, please stop your checks until the issue is resolved. | 11:15 |
| openstackstatus | yolanda: sending notice | 11:15 |
| -openstackstatus- NOTICE: Precise tests on OSIC provider are currently failing, please stop your checks until the issue is resolved. | 11:16 | |
| sdague | yolanda: it's more than just precise | 11:17 |
| sdague | there are 2 classes of issues | 11:17 |
| *** ccamacho is now known as ccamacho|lunch | 11:17 | |
| *** ManishD has quit IRC | 11:17 | |
| sdague | 1 is precise ipv6 seems to act up, which is only a system-config problem I think, but the ip collision issue is dsvm of all kinds on osic | 11:17 |
| robcresswell | precise tests? I don't suppose we could get a bit more info there could we? :) | 11:17 |
| sdague | that's the bigger fail | 11:17 |
| openstackstatus | yolanda: finished sending notice | 11:18 |
| yolanda | sdague, so it's all devstack on osic? do you have creds to send an alert as well, or can you suggest a better message? | 11:18 |
| sdague | yolanda: I do not know if I have creds or not | 11:19 |
| yolanda | you should | 11:19 |
| yolanda | i see you on the list | 11:19 |
| sdague | #status notice DSVM jobs on OSIC currently failing because of IP collisions, fix is in the gate - https://review.openstack.org/#/c/357764/ - please hold rechecks until merged | 11:20 |
| openstackstatus | sdague: sending notice | 11:20 |
| robcresswell | Ah, excellent, thankyou sdague, yolanda | 11:21 |
| yolanda | sdague thx | 11:21 |
| sdague | robcresswell: also Precise there was a noun not an adj | 11:21 |
| sdague | it meant ubuntu 12.04 | 11:21 |
| -openstackstatus- NOTICE: DSVM jobs on OSIC currently failing because of IP collisions, fix is in the gate - https://review.openstack.org/#/c/357764/ - please hold rechecks until merged | 11:21 | |
| robcresswell | sdague: D'oh. Understood. I thought it meant "specific". | 11:21 |
| yolanda | oh, i didn't think precise should have been misunderstood like that | 11:22 |
| rcarrillocruz | heh | 11:22 |
| yolanda | non-native english problems... | 11:22 |
| robcresswell | Might just be my slow brain. Since it was a capital at the start of a sentence, I didn't think to view it as a noun. | 11:22 |
| openstackstatus | sdague: finished sending notice | 11:23 |
| *** matbu is now known as matbu|lch | 11:23 | |
| openstackgerrit | Merged openstack-infra/system-config: Set compute015.vanilla IP addresses https://review.openstack.org/357093 | 11:25 |
| *** psanchez has quit IRC | 11:26 | |
| *** SotK has quit IRC | 11:26 | |
| *** waynr has quit IRC | 11:26 | |
| *** pkoniszewski has quit IRC | 11:26 | |
| odyssey4me | sdague so the OSIC cluster's IPv4 range is that big to match the IPV6 range... all instances are setup with a public IPV6 and a non-routable IPV4 | 11:26 |
| *** sarob has joined #openstack-infra | 11:26 | |
| sdague | so... dougwig ... I kind of think the situation had self cleared | 11:26 |
| odyssey4me | but yes, perhaps the range should be made just that little bit smaller to prevent collissions | 11:26 |
| odyssey4me | once cloudnull comes online we can figure it out - it should just be a neutron network range change | 11:27 |
| sdague | dougwig: we are now getting 10.1.20.x ips | 11:27 |
| *** SotK has joined #openstack-infra | 11:27 | |
| *** psanchez has joined #openstack-infra | 11:27 | |
| sdague | which would be outside the old fixed ip range | 11:27 |
| *** matthewbodkin has quit IRC | 11:27 | |
| *** pkoniszewski has joined #openstack-infra | 11:27 | |
| *** waynr has joined #openstack-infra | 11:27 | |
| dougwig | sdague: ha, i don't know whether to laugh or cry about that. | 11:27 |
| odyssey4me | IIRC the network range could remain, but the DHCP range on the network could be set to exclude certain blocks | 11:28 |
| sdague | dougwig: well, we aren't the only consumers of those IPs | 11:28 |
| *** ociuhandu has joined #openstack-infra | 11:28 | |
| sdague | odyssey4me: sure, it's just, you probably don't need that whole range right, because it's not like you are going to have that many machines alive at once | 11:28 |
| odyssey4me | sdague yeah, fair point | 11:28 |
| sdague | and the ipam will recycle unused once it hits the end of the range | 11:28 |
| odyssey4me | ok, I suppose the best solution here is to cut down the DHCP range to the size of the intended quota for cloud1 | 11:29 |
| dougwig | i want to see the hypervisors that can run 16M vm's. that'd be neat. | 11:29 |
| odyssey4me | sdague what're the ranges you want to exclude? | 11:29 |
| sdague | dougwig: I want to see network infrastructure that can do that | 11:30 |
| odyssey4me | ah, I see it - 10.1.0.0./20, then also 10.199.0.0/20 | 11:30 |
| *** esikachev has joined #openstack-infra | 11:30 | |
| dougwig | sdague: i'd bet neutron would explode and die. | 11:30 |
| sdague | odyssey4me: well, we could easily revert | 11:30 |
| dougwig | odyssey4me: 10/16 would avoid the issue. | 11:30 |
| sdague | odyssey4me: the important thing is just that the whole 10.0.0.0/8 isn't being allocated | 11:30 |
| *** sarob has quit IRC | 11:31 | |
| *** ramishra has quit IRC | 11:31 | |
| sdague | dougwig: right, still provides 64k vms in the cluster | 11:31 |
| *** lucasagomes is now known as lucas-hungry | 11:31 | |
| *** Na3iL has quit IRC | 11:32 | |
| *** jkilpatr has joined #openstack-infra | 11:32 | |
| *** markvoelker has joined #openstack-infra | 11:32 | |
| odyssey4me | based on the proposed quota of 512 nodes... I expect that anything /22 or larger will be just fine as a DHCP range | 11:32 |
| dougwig | sdague: i think you just got a valid comment on your d-g change, about the gateway ip | 11:33 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: Inject CA certificate via service profile https://review.openstack.org/357791 | 11:33 |
| *** ramishra has joined #openstack-infra | 11:33 | |
| sdague | dougwig: ug, right, neutron requires you synchronize those | 11:33 |
| sdague | well, it should fail | 11:33 |
| *** javeriak has joined #openstack-infra | 11:33 | |
| sdague | it's on my list to make that a sane default | 11:34 |
| amoralej | neutron jobs are failing, that's why i realized | 11:34 |
| sdague | amoralej: yeh | 11:34 |
| sdague | so... here is the thing, in looking at logstash, I think we're no longer allocating in the failed range | 11:35 |
| dougwig | then we can likely make this change leisurely, and not in crisis mode. | 11:35 |
| odyssey4me | sdague I expect they're all used, so until it circles back to there I guess we're passed the danger zone | 11:35 |
| sdague | yeh, or more importantly just get osic | 11:35 |
| sdague | to change to exclude some of this | 11:35 |
| *** dprince has joined #openstack-infra | 11:35 | |
| odyssey4me | the osic/cloud1 crew should be online shortly and can resolve the range used to something more reasonable | 11:36 |
| sdague | odyssey4me: cool, thanks | 11:36 |
| *** _ari_ has joined #openstack-infra | 11:36 | |
| *** ldnunes has joined #openstack-infra | 11:36 | |
| odyssey4me | well, by shortly I mean anywhere in the next 2-4 hours | 11:36 |
| sdague | dougwig: you don't have any recent runs (last hour) that fail, right? | 11:37 |
| *** markvoelker has quit IRC | 11:37 | |
| dougwig | sdague: i do, i assume we have some bad nodes in nodepool. let me try to a new recheck | 11:38 |
| *** ihrachys has quit IRC | 11:38 | |
| *** Na3iL has joined #openstack-infra | 11:39 | |
| dougwig | sdague: a spot check of zuul looks happy. | 11:40 |
| *** jaosorior has quit IRC | 11:41 | |
| *** jaosorior has joined #openstack-infra | 11:42 | |
| *** baoli has joined #openstack-infra | 11:42 | |
| eset | eh still the same errors at disk-image-create -> http://paste.openstack.org/raw/561274/ | 11:43 |
| *** thorst_ has joined #openstack-infra | 11:43 | |
| sdague | yolanda: do I just send another status to tell folks we're probably ok now? | 11:47 |
| yolanda | yes please | 11:47 |
| *** dizquierdo has joined #openstack-infra | 11:47 | |
| sdague | #status OSIC has burned through the problematic IP range with failures, things should be back to normal now. | 11:48 |
| openstackstatus | sdague: unknown command | 11:48 |
| sdague | #status notice OSIC has burned through the problematic IP range with failures, things should be back to normal now. | 11:48 |
| openstackstatus | sdague: sending notice | 11:48 |
| -openstackstatus- NOTICE: OSIC has burned through the problematic IP range with failures, things should be back to normal now. | 11:49 | |
| openstackstatus | sdague: finished sending notice | 11:51 |
| mordred | sdague: there's a review comment on the IP collision fix | 11:52 |
| openstackgerrit | Merged openstack-infra/project-config: Temporarily remove osic nodes for ubuntu-precise https://review.openstack.org/357742 | 11:53 |
| mordred | sdague: gah. sorry. I was scrolled too far back | 11:53 |
| sdague | mordred: yeh, so, we're actually already on the other side of the ip collision block | 11:54 |
| mordred | yup | 11:54 |
| sdague | tests failing in 13 minutes and a loaded gate helped there | 11:54 |
| mordred | I'm caught up now | 11:54 |
| sdague | I'm also going to fix devstack so that we don't have to set all these things manually, because it's silly | 11:54 |
| mordred | ++ | 11:55 |
| *** julim has quit IRC | 11:57 | |
| mordred | odyssey4me, sdague: so - reading more of the scrollback - the private network we have is not exclusive to our tenant. GATEWAY_NET_V6 gives us a public v6 and a private v4 | 11:58 |
| mordred | if you want ot see the exact actual networks and subnets in play, check out https://review.openstack.org/#/c/357517/1/shade/tests/unit/test_meta.py,unified | 11:58 |
| mordred | which includes copies of the existing networks and subnets for OSIC | 11:58 |
| mordred | (so that we can unittest we don't break) | 11:58 |
| *** amoralej is now known as amoralej|lunch | 11:58 | |
| mordred | GATEWAY_SUBNET_V6V4 is the one we currently get private v4 from because we boot on GATEWAY_NET_V6 | 11:59 |
| odyssey4me | mordred and the allocation range is where the problem lies: 10.255.255.254', u'start': u'10.0.0.2 | 12:00 |
| mordred | yah | 12:00 |
| odyssey4me | that overlaps with addresses used by various projects | 12:00 |
| odyssey4me | devstack: 10.1.0.0/20 | 12:00 |
| sdague | right, and devstack will not detect HOST_IP if it's within the FIXED or FLOATING ranges, because, things can go super weird | 12:00 |
| odyssey4me | OSA integrated: 10.255.255.0/24 | 12:01 |
| odyssey4me | OSA roles: 10.100.100.0/24 | 12:01 |
| odyssey4me | LXC default: 10.1.3.0/24 | 12:01 |
| odyssey4me | those are the ones I know about | 12:01 |
| mordred | yup. no, totally agree ... just pointing out that the problem space isn't "make the private net from osic the same size as the quota" | 12:02 |
| *** dmellado is now known as dmellado|lunch | 12:02 | |
| *** dmellado|lunch is now known as dmellado | 12:02 | |
| sdague | mordred: oh, yeh, agreed | 12:02 |
| *** zz_dimtruck is now known as dimtruck | 12:02 | |
| sdague | I just want it not the whole 10.x space | 12:02 |
| *** mfedosin has joined #openstack-infra | 12:02 | |
| odyssey4me | mordred sure, I was just suggesting that the allocation range be around the same size as the quota | 12:02 |
| sdague | because we need some breathing space | 12:02 |
| *** matbu|lch is now known as matbu | 12:02 | |
| mordred | odyssey4me: it can't be | 12:02 |
| mordred | odyssey4me: that subnet is not specific to us | 12:03 |
| sdague | mordred: right | 12:03 |
| sdague | but a /16 would be fine | 12:03 |
| odyssey4me | mordred heh, oh yes - that's the 'public' network for OSIC devs too | 12:03 |
| mordred | sdague: exactly | 12:03 |
| sdague | it's just the /8 that we're wandering across that's the problem | 12:03 |
| sdague | because we have no where to hide | 12:04 |
| odyssey4me | so yeah, even if /8 is used - the blocks used by other projects should be left out of the range | 12:04 |
| *** ihrachys has joined #openstack-infra | 12:04 | |
| sdague | odyssey4me: sure, ours is config though, so we can move it around | 12:04 |
| odyssey4me | same here | 12:04 |
| sdague | it's not even the devstack default, which is 10.0.0.0/24, because that used to conflict with HP cloud | 12:05 |
| odyssey4me | it might be a good idea for us to set some sort of standard for tests across projects | 12:05 |
| *** yamamoto has quit IRC | 12:05 | |
| sdague | odyssey4me: good luck with that :) | 12:05 |
| odyssey4me | that way we can ensure that no nodepool providers clash with the dhcp ranges provided to nodepool images | 12:05 |
| sdague | I think in reality osic should just declare a range smaller than /8, and document it | 12:05 |
| sdague | and let everyone else move around in the freespace as they will | 12:06 |
| *** yamamoto has joined #openstack-infra | 12:06 | |
| *** pgadiya has quit IRC | 12:06 | |
| odyssey4me | 10.10.0.0/16 will avoid the currently known clashes | 12:06 |
| *** rodrigods has quit IRC | 12:08 | |
| *** rodrigods has joined #openstack-infra | 12:08 | |
| sdague | sure | 12:08 |
| odyssey4me | mordred not sure if you've seen this? https://review.openstack.org/356241 | 12:08 |
| *** edmondsw has joined #openstack-infra | 12:08 | |
| mordred | oh - I haven't. one sec | 12:08 |
| *** rfolco has joined #openstack-infra | 12:10 | |
| *** yamamoto has quit IRC | 12:11 | |
| *** dimtruck is now known as zz_dimtruck | 12:11 | |
| *** ccamacho|lunch is now known as ccamacho | 12:12 | |
| *** zhurong has joined #openstack-infra | 12:16 | |
| *** tpsilva has joined #openstack-infra | 12:16 | |
| sdague | can we get reviews on this - https://review.openstack.org/#/c/357443/ - d-g change so that we can drop the pg job later and not stop testing metadata server | 12:19 |
| *** pradk has joined #openstack-infra | 12:22 | |
| *** raildo has joined #openstack-infra | 12:23 | |
| *** gordc has joined #openstack-infra | 12:24 | |
| *** asettle has quit IRC | 12:26 | |
| *** javeriak has quit IRC | 12:27 | |
| *** matthewbodkin has joined #openstack-infra | 12:27 | |
| *** mordred_ has joined #openstack-infra | 12:27 | |
| *** asettle has joined #openstack-infra | 12:28 | |
| mordred_ | laptop exploded - I may be fairly useless for a few hours | 12:28 |
| *** esikachev has quit IRC | 12:29 | |
| *** mordred_ is now known as sadmordred | 12:31 | |
| *** rhallisey has joined #openstack-infra | 12:32 | |
| *** markvoelker has joined #openstack-infra | 12:33 | |
| *** sdake has joined #openstack-infra | 12:35 | |
| *** rhallisey_ has joined #openstack-infra | 12:35 | |
| *** yamamoto has joined #openstack-infra | 12:36 | |
| *** markvoelker has quit IRC | 12:38 | |
| *** alaski is now known as lascii | 12:38 | |
| *** rhallisey has quit IRC | 12:39 | |
| *** javeriak has joined #openstack-infra | 12:40 | |
| *** lucas-hungry is now known as lucasagomes | 12:41 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: Set Mistral workers to 1 https://review.openstack.org/356375 | 12:42 |
| openstackgerrit | Merged openstack-infra/groups-static-pages: Remove groups.json https://review.openstack.org/356843 | 12:43 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: Set Sahara workers to 1 https://review.openstack.org/352729 | 12:43 |
| *** rhallisey_ is now known as rhallisey | 12:47 | |
| *** jtomasek has joined #openstack-infra | 12:48 | |
| openstackgerrit | Merged openstack-infra/groups: Group contanct report csv dump filter by group status https://review.openstack.org/357175 | 12:49 |
| *** rlandy has joined #openstack-infra | 12:49 | |
| eset | umount: /tmp/dib_build.IHCiaBWV: device is busy. | 12:49 |
| eset | eh still the same | 12:49 |
| *** markvoelker has joined #openstack-infra | 12:49 | |
| *** ihrachys has quit IRC | 12:50 | |
| *** ihrachys has joined #openstack-infra | 12:50 | |
| *** baoli has quit IRC | 12:51 | |
| *** gouthamr has joined #openstack-infra | 12:52 | |
| *** javeriak has quit IRC | 12:54 | |
| *** kushal has joined #openstack-infra | 12:56 | |
| *** salv-orlando has quit IRC | 12:57 | |
| *** dprince has quit IRC | 12:57 | |
| *** matt-borland has joined #openstack-infra | 13:00 | |
| *** pilgrimstack has quit IRC | 13:01 | |
| *** kushal has quit IRC | 13:01 | |
| *** dprince has joined #openstack-infra | 13:02 | |
| *** fguillot has joined #openstack-infra | 13:02 | |
| *** baoli has joined #openstack-infra | 13:03 | |
| *** dizquierdo has quit IRC | 13:03 | |
| *** kushal has joined #openstack-infra | 13:04 | |
| *** Julien-zte has joined #openstack-infra | 13:04 | |
| *** andymaier has joined #openstack-infra | 13:05 | |
| *** Na3iL has quit IRC | 13:06 | |
| *** nmagnezi has joined #openstack-infra | 13:06 | |
| *** xyang1 has joined #openstack-infra | 13:07 | |
| *** fifieldt has quit IRC | 13:07 | |
| *** coolsvap is now known as _coolsvap_ | 13:08 | |
| *** jtomasek has quit IRC | 13:11 | |
| *** amoralej|lunch is now known as amoralej | 13:12 | |
| *** kushal has quit IRC | 13:12 | |
| *** asettle has quit IRC | 13:13 | |
| *** mtanino has joined #openstack-infra | 13:13 | |
| *** asettle has joined #openstack-infra | 13:14 | |
| openstackgerrit | Matthew Bodkin proposed openstack-infra/storyboard-webclient: Make sidebar submenu the same length as sidebar https://review.openstack.org/355554 | 13:15 |
| *** fguillot has quit IRC | 13:16 | |
| *** fifieldt has joined #openstack-infra | 13:18 | |
| *** kgiusti has joined #openstack-infra | 13:19 | |
| *** jianghuaw has quit IRC | 13:20 | |
| *** kushal has joined #openstack-infra | 13:21 | |
| openstackgerrit | Merged openstack-infra/groups: Security update for Google Analytics, Panels, Panelizer https://review.openstack.org/357340 | 13:21 |
| *** mtanino has quit IRC | 13:21 | |
| *** mdrabe has joined #openstack-infra | 13:22 | |
| *** jianghuaw has joined #openstack-infra | 13:23 | |
| pabelanger | morning | 13:23 |
| *** mriedem has joined #openstack-infra | 13:24 | |
| asselin_ | rcarrillocruz, you around to discuss cloud-launcher? | 13:24 |
| cloudnull | mornings | 13:26 |
| jpmaxman | fungi: curious about the wiki puppet scripts supporting Precise. More for my own understanding than anything. It looks to me like the puppet scripts are set to check out the latest upstream mediawiki repository. But the latest (1.27) doesn't run on precise. So, instead of supporting precise shouldn't it prevent precise? | 13:27 |
| odyssey4me | cloudnull would it be possible to reduce the DHCP allocation range for OSIC cloud1 to something like 10.10.0.0/16 to avoid conflicts with devstack (10.1.0.0/20), OSA (10.255.255.0/24, 10.100.100.0/24) ? | 13:29 |
| cloudnull | sdague odyssey4me: what clashes do we need to deal with ? | 13:29 |
| odyssey4me | ^ | 13:29 |
| cloudnull | oh. | 13:29 |
| cloudnull | sure. | 13:29 |
| odyssey4me | alternatively, could we just exclude those ranges from the allocation pool | 13:30 |
| sdague | cloudnull: yeh, just anything smaller than /8 | 13:30 |
| fungi | jpmaxman: yeah, since we're not actually running on precise any longer, i decided to ignore that and we'll just clean up the precise vs trusty logic in there later | 13:30 |
| sdague | we can work around anything where we have some space in the 10.x range that doesn't overlap | 13:30 |
| *** jheroux has joined #openstack-infra | 13:31 | |
| sdague | but the current allocation assumes it owns all of /8, and it overran a range we use for testing, causing fails | 13:31 |
| *** julim has joined #openstack-infra | 13:31 | |
| sdague | it failed enough, fast enough, that it consumed all the way to the other end of the conflict range in 24 hrs | 13:31 |
| sdague | however, still would rather avoid | 13:31 |
| odyssey4me | something we could simply agree on is that we all use something like 10.255.254.0/24 in tests to avoid clashes with typical allocations in nodepool providers | 13:31 |
| *** kushal has quit IRC | 13:31 | |
| openstackgerrit | Merged openstack-infra/system-config: Add source to ubuntu mirror distributions https://review.openstack.org/357560 | 13:32 |
| fungi | sdague: i'm sure it was already considered but i didn't see it... why can't we just down the eth1 interfaces? we don't need to communicate through them anyway | 13:32 |
| cloudnull | have a set range that projects can use for testing would be great | 13:32 |
| JerryOpenix | Hi all, I want to know who has the rights to support "workflow"? | 13:32 |
| sdague | right, well here is the thing. The cloud providers do what they do for reasons before they meet us | 13:33 |
| fungi | JerryOpenix: i have no idea what that means. what is "workflow"? | 13:33 |
| sdague | so I don't think we should ever assume there is a "test safe" range | 13:33 |
| jpmaxman | fungi: yeah that's fine and obviously whatever is the quickest path to a decent puppet build. was more asking for clarity on process. typically you'd want to prevent precise it seems? is that right? I mean is it possible to prevent? | 13:34 |
| cloudnull | so maybe devstack and osa should fall back to a secondary range when the primary is consumed? then you'd fense against these types of collisions? | 13:34 |
| odyssey4me | sdague cloudnull an alternative would be to do some sort of detection and to dynamically set the test address CIDT based on one the host doesn't have in its routing table | 13:34 |
| odyssey4me | *CIDR | 13:34 |
| *** raunak has joined #openstack-infra | 13:34 | |
| *** baoli has quit IRC | 13:34 | |
| JerryOpenix | "workflow" is one of the steps for merging patches. | 13:35 |
| cloudnull | I'm happy to change the range, I'm juts pointing out that its likely this will happen again. | 13:35 |
| cloudnull | not just in the osic | 13:35 |
| odyssey4me | that would probably be the only *safe* way to do it | 13:35 |
| fungi | jpmaxman: we choose what distro we're installing on, but for the benefit of other consumers we could add a check to print a warning if the module is applied to a distro/release that isn't one we expect that module to support (we do that elsewhere in other modules already) | 13:36 |
| sdague | odyssey4me: the more complicated you make this, the more likely it will fail | 13:36 |
| odyssey4me | sdague yeah, which is why we haven't done it | 13:36 |
| *** jcoufal has joined #openstack-infra | 13:36 | |
| odyssey4me | we opted for the least complicated way - just change the CIDR for the tests and move on | 13:37 |
| sdague | really, the issue is that osic claims the entirey 10.0.0.0/8 range | 13:37 |
| sdague | for no real reason | 13:37 |
| sdague | no other provider does that | 13:37 |
| odyssey4me | well, it may happen as others go to IPV6 | 13:37 |
| cloudnull | ^ it might | 13:38 |
| odyssey4me | but this is a lesson learned which we can carry forward | 13:38 |
| sdague | odyssey4me: I don't understand why that's believed | 13:38 |
| fungi | JerryOpenix: oh, you're talking about the workflow label in gerrit. that is determined by project-specific acls. workflow -1..0 is granted globally to anyone with an account in gerrit, but workflow -1..+1 is typically granted only to a project-specific core review group | 13:38 |
| jpmaxman | fungi: ok but from an internal process you mostly just trust the admin to know what distro and wouldn't worry about config file "if/then" lines to cover those previous distros that weren't supported? | 13:38 |
| *** esberglu has joined #openstack-infra | 13:38 | |
| fungi | jpmaxman: basically, yes. i expect one of the coming patches for that module will just drop the release-specific conditional logic, at least until someone starts to work on porting it to xenial | 13:39 |
| fungi | which we're in no hurry for | 13:39 |
| cloudnull | sdague: RAX consume most, if not all, of the 10.x space for the service net. they're just broken out into cells/huddles so maybe you dont notice the conflicts as frequently. | 13:41 |
| *** bhunter71 has joined #openstack-infra | 13:41 | |
| cloudnull | 's/./?/' | 13:42 |
| fungi | sdague: i'm sure it was already considered but i didn't see it... why can't we just down the eth1 interfaces? we don't need to communicate through them anyway, and i believe some of our providers don't even give us a second interface | 13:42 |
| *** derekh has quit IRC | 13:42 | |
| cloudnull | dont get me wrong, i'm making the change. | 13:42 |
| *** _ari_ has quit IRC | 13:42 | |
| sdague | fungi: this isn't about the second interface | 13:42 |
| fungi | sdague: oh, this is on eth0? | 13:42 |
| sdague | this is detecting HOST_IP | 13:42 |
| sdague | which we exclude the fixed and floating ranges | 13:42 |
| *** dprince has quit IRC | 13:43 | |
| *** hongbin has joined #openstack-infra | 13:43 | |
| fungi | yeah, that complicates matters if this is on eth0 | 13:43 |
| sdague | because that causes chaos if you don't get things right | 13:43 |
| sdague | yes, this is the external interface for the box | 13:43 |
| *** dprince has joined #openstack-infra | 13:43 | |
| sdague | it's the address nodepool and everything is talking to | 13:43 |
| jpmaxman | yup I get it - just more trying to understand holistically and be able to provide better reviews. The wiki is a little odd as there were subtle config file conventions changed between 1.25--> 1.27 some of which seem to be backward compatible and others of which are not. So moving forward with the puppet manifests it's a bit difficult if it is checking out | 13:43 |
| jpmaxman | the upstream repository and they happen to change something. But I guess that's just a fact of life you have to deal with and be ready to patch the puppet accordingly. | 13:43 |
| sdague | cloudnull: so, it at least seems that our tenants in rax are getting 10.208/19 and 10.210/19 network blocks | 13:44 |
| *** _ari_ has joined #openstack-infra | 13:44 | |
| *** acabot has quit IRC | 13:45 | |
| *** derekh has joined #openstack-infra | 13:45 | |
| fungi | sdague: and the shuffling logic devstack uses to pick an available cidr only allocates from somewhere in 10/8, can't switch to something in 172.16/12 or 192.168/16 | 13:45 |
| sdague | fungi: devstack doesn't pick a cidr | 13:46 |
| *** zul has quit IRC | 13:46 | |
| fungi | oh, it's a hard-coded cidr? | 13:46 |
| sdague | yes | 13:46 |
| fungi | mmm, hard cidr | 13:46 |
| sdague | and d-g has a different one | 13:46 |
| *** bhunter71 has quit IRC | 13:46 | |
| sdague | the failure is about detecting the HOST_IP, which is needed for service addresses and the like | 13:46 |
| *** nwkarsten has joined #openstack-infra | 13:46 | |
| sdague | and the logic to detect ones assumes that you can't have meant for it to be inside your fixed or floating ranges | 13:47 |
| sdague | because, some weird stuff can happen | 13:47 |
| sdague | then it finds no valid HOST_IP candidates and fails | 13:47 |
| sdague | and we do dynamic detection of *that* for IPv6 | 13:47 |
| *** Ravikiran_K has quit IRC | 13:47 | |
| *** baoli has joined #openstack-infra | 13:47 | |
| *** nmagnezi has quit IRC | 13:47 | |
| sdague | because if d-g or others hard coded ipv4 address, it would break ipv6 only test envs | 13:47 |
| sdague | which do exist | 13:48 |
| odyssey4me | yeah, I'm quite sure that a bunch of tests use 172.16.0.0/16 and there are possibly also 192.168.0.0/24 allocations too | 13:48 |
| fungi | jpmaxman: what i'm picturing is that we'll end up pinning to the 1.27 series in production, but can try running master on the -dev host and when it ceases to be compatible with the rest of the manifest we start implementing version-specific logic so that when we're ready to raise the version cap in production it will be compatible | 13:48 |
| sdague | odyssey4me: 172.16.0.0/16 is used for the floating ip blocks | 13:48 |
| odyssey4me | sdague yeah, we do that too for tempest tests | 13:48 |
| *** dansmith is now known as superdan | 13:48 | |
| sdague | and 192.168.x was avoided because consumer router defaults | 13:48 |
| odyssey4me | ah, good point | 13:49 |
| sdague | and vm envs like virtualbox and kvm | 13:49 |
| *** esikachev has joined #openstack-infra | 13:50 | |
| *** vsaienko has joined #openstack-infra | 13:51 | |
| sdague | so, yes, our address selection is heuristic based on least likely to fail. That heuristic broke down here. We could make d-g choose some other range IIF it is on osic, but it seems like that /8 is going to hurt other folks as well, and seems way larger than is needed, and the people running that cloud are here :) | 13:51 |
| *** Na3iL has joined #openstack-infra | 13:52 | |
| vsaienko | infra-cores, please review https://review.openstack.org/#/c/340295/ We have an idea how to decrease number of ironic jobs, and put them all together. | 13:52 |
| *** thiagop has joined #openstack-infra | 13:52 | |
| *** zul has joined #openstack-infra | 13:53 | |
| *** woodster_ has joined #openstack-infra | 13:53 | |
| cloudnull | so it would seem neutron has no ability to change a CIDR via the CLIs which means I have to update the record in the DB. | 13:54 |
| fungi | yeah, even if they halved it to 10/9 or 10.128/9 that's still 2^23 addresses | 13:54 |
| cloudnull | ^ unless someone knows the command to make that go ? | 13:54 |
| jpmaxman | fungi: cool - there's also the argument that it's better to break it than let it sit and not be upgraded :) either way... again more for my knowledge than anything. appreciate the explanations. | 13:54 |
| cloudnull | i can adjust the allocation pool w/ ease | 13:54 |
| *** e0ne_ has joined #openstack-infra | 13:55 | |
| *** burgerk has joined #openstack-infra | 13:55 | |
| *** e0ne has quit IRC | 13:55 | |
| fungi | jpmaxman: right, i think with a dev server consuming master and a production server pinned to a known working version we sort of get both. dev will spontaneously break when backward compatibility ceases, but production is only impacted on our timetable | 13:56 |
| *** piet has joined #openstack-infra | 13:57 | |
| *** slaweq_ has quit IRC | 13:57 | |
| odyssey4me | cloudnull yeah, just adjust the allocation pool - not the CIDR | 13:57 |
| jpmaxman | fungi: yup CI to dev - controlled pushes to production - that's kind of how we roll too - sounds good | 13:57 |
| odyssey4me | that will allow you to change it again later if you need to | 13:57 |
| odyssey4me | cloudnull that *should* be an online change too which won't cause any existing VM's to go offline | 13:58 |
| cloudnull | and we need to avoid 10.255.255.0/24, 10.100.100.0/24, 10.1.0.0/20 ? | 13:58 |
| odyssey4me | cloudnull yes, those are the ones we know of right now | 13:59 |
| cloudnull | allocation pool is an online change, CIDR change will need to be done in a maint window | 13:59 |
| *** andymaier has quit IRC | 13:59 | |
| *** acabot has joined #openstack-infra | 13:59 | |
| cloudnull | so i'm just doing the pool modifications and wont worry about actually changing the CIDR. | 14:00 |
| fungi | sdague: just avoiding having the assigned addresses land in the bounds of a problem cidr is sufficient to avoid the problem right? if so, allocation pool ought to be sufficient | 14:00 |
| sdague | fungi: yep | 14:01 |
| sdague | yeh, all we need is for our host address to not be in that range | 14:01 |
| sdague | honestly, because of the burn rate, we're probably safe for a few months anyway, as it has to loop all the way around | 14:02 |
| sdague | but it would be good to think about this long term | 14:02 |
| *** zz_dimtruck is now known as dimtruck | 14:02 | |
| pabelanger | fungi: cloudnull: I retested our initialize-urandom changes manually on an existing ubuntu-xenial image in osic-cloud1, and it worked. They were merged this morning by yolanda, so today ubuntu-xenail DIB should work properly | 14:04 |
| cloudnull | nice! | 14:04 |
| cloudnull | sdague: in the next maint window i'll have it adjusted. | 14:05 |
| pabelanger | fungi: I'm uploading ubuntu-precise right now also, that should address our network issues we see in osic-cloud1 | 14:05 |
| pabelanger | since ipv6 privacy is now disbled | 14:05 |
| sdague | cloudnull: awesome | 14:06 |
| sdague | cloudnull: thanks much | 14:06 |
| fungi | thanks pabelanger | 14:06 |
| cloudnull | sorry for the hassel . | 14:06 |
| fungi | thanks cloudnull! | 14:06 |
| sdague | it was a sufficiently odd debug this morning to figure out what was going on. But kind of a fun puzzle :) | 14:06 |
| *** eandersson_ has quit IRC | 14:07 | |
| *** eandersson_ has joined #openstack-infra | 14:08 | |
| *** rbrndt has joined #openstack-infra | 14:09 | |
| zxiiro | electrofelix: #openstack-sprint ? | 14:10 |
| *** dimtruck is now known as zz_dimtruck | 14:12 | |
| cloudnull | pools updated. | 14:13 |
| *** chlong has joined #openstack-infra | 14:13 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: Use separated SSL endpoint environment file https://review.openstack.org/356488 | 14:14 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: Inject CA certificate via service profile https://review.openstack.org/357791 | 14:14 |
| pabelanger | fungi: and uploaded. let me see if I can trigger a build on osic | 14:17 |
| *** hpe-hj has joined #openstack-infra | 14:17 | |
| *** yamahata has joined #openstack-infra | 14:18 | |
| *** ihrachys has quit IRC | 14:18 | |
| electrofelix | zxiiro: yep, thanks, was just afk for a little while there | 14:19 |
| *** wznoinsk has quit IRC | 14:19 | |
| *** camunoz has joined #openstack-infra | 14:19 | |
| *** matt-borland has quit IRC | 14:20 | |
| *** raunak has quit IRC | 14:21 | |
| pabelanger | nc 2001:4800:1ae1:18:f816:3eff:fe30:7b72 19885 | 14:21 |
| pabelanger | is our ubuntu-precise job | 14:21 |
| *** ihrachys has joined #openstack-infra | 14:21 | |
| pabelanger | looks to be fast | 14:21 |
| *** afred312_ has quit IRC | 14:21 | |
| pabelanger | sorry | 14:21 |
| pabelanger | that is debian-jessie | 14:21 |
| *** raunak has joined #openstack-infra | 14:22 | |
| *** annegentle has joined #openstack-infra | 14:22 | |
| *** afred312 has joined #openstack-infra | 14:23 | |
| apuimedo | pabelanger: how do we get new groups created in gerrit? | 14:26 |
| apuimedo | we'd need a kuryr-libetnwork-core and kuryr-kubernetes-core | 14:26 |
| *** jtomasek has joined #openstack-infra | 14:27 | |
| *** zz_dimtruck is now known as dimtruck | 14:28 | |
| *** jtomasek has quit IRC | 14:28 | |
| *** mtanino has joined #openstack-infra | 14:28 | |
| *** jtomasek has joined #openstack-infra | 14:29 | |
| *** valderrv_ has joined #openstack-infra | 14:29 | |
| *** raunak has quit IRC | 14:29 | |
| *** valderrv_ has quit IRC | 14:29 | |
| fungi | apuimedo: reference them in gerrit project acls | 14:29 |
| *** wznoinsk has joined #openstack-infra | 14:30 | |
| fungi | they'll be automatically created when the acl updates are applied, and then you just need to ping one of our gerrit admins to add your ptl as the initial group member | 14:30 |
| apuimedo | oh | 14:30 |
| apuimedo | ok | 14:30 |
| apuimedo | let's try | 14:30 |
| pabelanger | apuimedo: additional info a top of what fungi said. http://docs.openstack.org/infra/manual/creators.html | 14:30 |
| apuimedo | thanks a lot fungi, pabelanger ! | 14:31 |
| pabelanger | look at the gerrit permissions section | 14:31 |
| *** _nadya_ has quit IRC | 14:31 | |
| fungi | yes, documentation. we have that! ;) | 14:31 |
| *** inc0 has joined #openstack-infra | 14:32 | |
| cloudnull | fungi: to circle back on the xenial changes, was it just a permissions issue on the unit file? | 14:32 |
| cloudnull | just to feed my own morbid curiosity, | 14:33 |
| pabelanger | think so, we just made changes to make sure we have the proper permissions on everything now | 14:33 |
| fungi | cloudnull: it ran deeper, but i fell asleep. pabelanger would know | 14:34 |
| pabelanger | I just kicked off an image build on nodepool, and going to test it first before uploading it to osic-cloud1 | 14:34 |
| pabelanger | but when I manually apply the changes to an existing image in osic-cloud1, it worked | 14:34 |
| pabelanger | unless it has to do with a first boot scenario that we some how missed | 14:35 |
| apuimedo | fungi: when you say reference tehm in gerrit project acls | 14:36 |
| apuimedo | you mean in gerrit https://review.openstack.org/#/admin/projects/openstack/kuryr-libnetwork,access | 14:36 |
| apuimedo | right? | 14:36 |
| pabelanger | apuimedo: you'd add new ACL files to: http://git.openstack.org/cgit/openstack-infra/project-config/tree/gerrit/projects.yaml#n3420 | 14:38 |
| apuimedo | pabelanger: ah, ok | 14:38 |
| apuimedo | sending the patch then | 14:38 |
| *** dimtruck is now known as zz_dimtruck | 14:38 | |
| pabelanger | you can remove the existing acl-config for both project and create kuryr-kubernetes.config and kuryr-libnetwork.config | 14:39 |
| *** tonytan4ever has joined #openstack-infra | 14:40 | |
| mriedem | are the devstack-gate-cleanup failures a known thing? | 14:41 |
| *** slaweq_ has joined #openstack-infra | 14:41 | |
| mriedem | oh nvm | 14:42 |
| *** asettle has quit IRC | 14:42 | |
| jaosorior | Can you guys point me to a project that is already using bindep? Would like to take something as a reference to take it into use myself | 14:44 |
| pabelanger | jaosorior: https://review.openstack.org/#/q/+topic:bindep-mv should be all the project currently using it | 14:47 |
| pabelanger | give you a good sample | 14:47 |
| fungi | though some early adopters may predate those changes | 14:47 |
| *** asselin__ has joined #openstack-infra | 14:47 | |
| jaosorior | pabelanger: thanks! | 14:48 |
| *** amitgandhinz has quit IRC | 14:49 | |
| *** amitgandhinz has joined #openstack-infra | 14:50 | |
| openstackgerrit | Antoni Segura Puimedon proposed openstack-infra/project-config: kuryr-libnetwork-core addition https://review.openstack.org/357945 | 14:50 |
| openstackgerrit | Antoni Segura Puimedon proposed openstack-infra/project-config: kuryr-kubernetes-core addition https://review.openstack.org/357946 | 14:50 |
| zigo | pabelanger: After merging a .gitreview file, the packages aren't published, and I can't find the log. For example, look at deb-python-concurrent.futures | 14:50 |
| zigo | pabelanger: http://logs.openstack.org/88/888f9de5ef010e29499041591cf68317629fbb11/ <--- 404 | 14:50 |
| apuimedo | fungi: pabelanger: thanks for the pointers. Sent in two patches above ^^ | 14:50 |
| zigo | pabelanger: Any idea what we've been doing wrong? | 14:51 |
| *** asselin_ has quit IRC | 14:51 | |
| pabelanger | zigo: when did you do the builds? | 14:51 |
| zigo | pabelanger: Oh, right, maybe I should wait... | 14:52 |
| zigo | pabelanger: hum... it's not in the zuul queue ... | 14:52 |
| *** matt-borland has joined #openstack-infra | 14:54 | |
| pabelanger | zigo: you are missing post pipelines for the projects | 14:54 |
| fungi | zigo: looking in zuul/layout.yaml that project only does merge-check and build-deb-package | 14:54 |
| *** burgerk has quit IRC | 14:54 | |
| zigo | Oh... | 14:54 |
| zigo | That's what I thought. | 14:54 |
| fungi | zigo: and the expansion of build-deb-package is just running pkgdeb-build-pkg in the check and gate pipelines | 14:55 |
| cloudnull | sdague: do we need to account for https://review.openstack.org/#/c/357764/1/devstack-vm-gate.sh@36 as well? | 14:55 |
| fungi | there's no post job to build and publish on merge events | 14:55 |
| openstackgerrit | Steve Martinelli proposed openstack-infra/project-config: setup translation job for keystonemiddleware https://review.openstack.org/357949 | 14:55 |
| pabelanger | zigo: fungi: Yup, these will be for jessie-newton right? | 14:55 |
| zigo | pabelanger: Right. | 14:55 |
| zigo | Looks like we need another infra patch... | 14:55 |
| pabelanger | zigo: yes, you need to add the post pipeline as deb-openstack-pkg-tools has to build-deb-package | 14:56 |
| *** vinaypotluri has joined #openstack-infra | 14:56 | |
| *** slaweq_ has quit IRC | 14:57 | |
| *** mdrabe has quit IRC | 14:58 | |
| openstackgerrit | Thomas Goirand proposed openstack-infra/project-config: Add post job after building deb packages https://review.openstack.org/357952 | 14:58 |
| zigo | pabelanger: Like that? :) | 14:58 |
| *** zz_dimtruck is now known as dimtruck | 14:59 | |
| pabelanger | zigo: almost | 15:00 |
| fungi | wow, i'm having serious problems syncing changes with gertty this morning. i wonder if something's causing performance issues for my uplink, or maybe my v6 tunnel | 15:00 |
| pabelanger | comment left | 15:00 |
| *** hockeynut has joined #openstack-infra | 15:00 | |
| *** salv-orlando has joined #openstack-infra | 15:01 | |
| openstackgerrit | Thomas Goirand proposed openstack-infra/project-config: Add post job after building deb packages https://review.openstack.org/357952 | 15:01 |
| *** dimtruck is now known as zz_dimtruck | 15:01 | |
| zigo | pabelanger: Should be good now then. | 15:01 |
| *** hockeynut has quit IRC | 15:02 | |
| zigo | pabelanger: How does your script know what repo to publish to? (ie: jessie-newton vs jessie-newton-backports) | 15:02 |
| zigo | Does it look at Distribution: in the .changes file? | 15:03 |
| *** zhurong has quit IRC | 15:03 | |
| pabelanger | zigo: the job template in JJB extras the info from job name | 15:03 |
| zigo | (ie: reprepro would do the work...) | 15:03 |
| pabelanger | reprepro-import-debian-openstack-jessie vs reprepro-import-debian-openstack-jessie-backports | 15:03 |
| *** mdrabe has joined #openstack-infra | 15:04 | |
| zigo | Ok. | 15:04 |
| pabelanger | we then look at the zuul branch to figure out which codename to use | 15:04 |
| zigo | :) | 15:04 |
| zigo | fungi: Can you +2 https://review.openstack.org/#/c/357952/ ? | 15:05 |
| *** zz_dimtruck is now known as dimtruck | 15:05 | |
| fungi | zigo: yeah, let's see if restarting gertty worked | 15:05 |
| fungi | much better! ;) | 15:05 |
| fungi | that may be our longest job dependency chain yet. nice | 15:06 |
| openstackgerrit | Merged openstack-infra/groups: Cleanup Drupal 7.50 install.inc patches https://review.openstack.org/357692 | 15:06 |
| *** _ari_ has quit IRC | 15:07 | |
| *** salv-orlando has quit IRC | 15:08 | |
| zigo | pabelanger: How can I remove dpkg, which I backported, but then decided it was a bad idea? | 15:08 |
| zigo | (ie: that's published in jessie-newton-backports) | 15:08 |
| *** dprince has quit IRC | 15:09 | |
| zigo | (FYI, I decided live-patching libdpkg-perl was best option, and that was needed to ignore changes in .gitreview files, otherwise git-buildpackage complains and fail) | 15:09 |
| zigo | In general, I believe I'll need a facility to remove stuff I pushed, somehow. | 15:10 |
| pabelanger | zigo: Hmm, that doesn't exist today. I'm not sure how to automate that ATM | 15:12 |
| pabelanger | zigo: obviously we can run commands manually on reprepro but don't want to do that too much | 15:12 |
| apuimedo | fungi: pabelanger: we need one more core to review https://review.openstack.org/#/c/346717/ | 15:13 |
| openstackgerrit | Thanh Ha proposed openstack-infra/jenkins-job-builder: Fix disabled always returning true https://review.openstack.org/357491 | 15:13 |
| openstackgerrit | Thanh Ha proposed openstack-infra/jenkins-job-builder: Add convenience function for plugin namespace https://review.openstack.org/357960 | 15:15 |
| zaro | morning | 15:15 |
| *** rbuzatu has quit IRC | 15:16 | |
| *** valderrv has joined #openstack-infra | 15:18 | |
| Krenair | fungi, pleia2: Hey, FYI: https://lists.wikimedia.org/pipermail/mediawiki-announce/2016-August/000194.html | 15:18 |
| pabelanger | zigo: we'd have to create a reprepro-remove-debian-openstack-jessie job for example, which took import from something, like a file. You'd then add each packaging you want to remove and merge it | 15:18 |
| *** netsinn has joined #openstack-infra | 15:18 | |
| pabelanger | zigo: maybe mordred has a better idea | 15:18 |
| openstackgerrit | Alex Schultz proposed openstack-infra/project-config: Update puppet integration jobs to use bindep.txt https://review.openstack.org/355203 | 15:18 |
| zigo | Ok. | 15:18 |
| fungi | Krenair: thanks for the heads-up! i'll plan to upgrade production to 1.27.1 on monday | 15:19 |
| Krenair | np. sorry I wasn't able to tell you earlier :) | 15:20 |
| *** _ari_ has joined #openstack-infra | 15:20 | |
| *** JerryOpenix has quit IRC | 15:21 | |
| *** markusry has joined #openstack-infra | 15:21 | |
| fungi | Krenair: no worries. i'm on the vmt for openstack, so i know how that goes | 15:21 |
| *** larainema has quit IRC | 15:21 | |
| *** netsin has quit IRC | 15:21 | |
| *** larainema has joined #openstack-infra | 15:23 | |
| *** rcernin has quit IRC | 15:24 | |
| *** _nadya_ has joined #openstack-infra | 15:24 | |
| fungi | i expect i'll see the vulnerability announcement come across the oss-security ml on monday anyway, which should be plenty of a reminder | 15:24 |
| *** piet has quit IRC | 15:24 | |
| *** jed56 has joined #openstack-infra | 15:26 | |
| *** pcaruana has quit IRC | 15:26 | |
| openstackgerrit | Ryan Brandt proposed openstack-infra/project-config: Remove Neutron tests from Monasca job https://review.openstack.org/357969 | 15:28 |
| *** ccamacho is now known as ccamacho|away | 15:29 | |
| openstackgerrit | Merged openstack-infra/project-config: Add post job after building deb packages https://review.openstack.org/357952 | 15:31 |
| *** armax has joined #openstack-infra | 15:31 | |
| *** gyee has joined #openstack-infra | 15:31 | |
| *** Swami has joined #openstack-infra | 15:31 | |
| clarkb | pabelanger: did xenial end uo building last night? | 15:32 |
| pabelanger | clarkb: yes, however we had issues with the image. We could not SSH into the server | 15:32 |
| _ari_ | weshay: ping | 15:32 |
| pabelanger | clarkb: we landed a potential fix over night and just finishing up the rebuild now | 15:33 |
| weshay | _ari_, pong | 15:33 |
| clarkb | pabelanger: what was the ssh problem? | 15:33 |
| pabelanger | clarkb: I believe we broke systemd stopping sshd from running | 15:33 |
| clarkb | wow nice | 15:33 |
| *** mdrabe has quit IRC | 15:33 | |
| pabelanger | yup | 15:33 |
| *** Goneri has quit IRC | 15:33 | |
| *** mdrabe has joined #openstack-infra | 15:33 | |
| cloudnull | pabelanger: is that still ongoing? anything i can help out w/ in troublshooting? | 15:34 |
| pabelanger | should have a new ubuntu-xenial image here in 30mins, plan to upload manually to osic-cloud for some testing, before having nodepool do it | 15:34 |
| *** xarses has joined #openstack-infra | 15:34 | |
| _ari_ | weshay: Hi kb gave us credentials to access the openstack instance in Centos do you know anything about that instance | 15:34 |
| pabelanger | cloudnull: We'll have more info in about 30 mins | 15:34 |
| cloudnull | ok | 15:34 |
| *** tonytan_brb has joined #openstack-infra | 15:34 | |
| weshay | _ari_, dmsimard is the only one familiar w/ it afaik | 15:34 |
| _ari_ | weshay: cool I will ping him | 15:35 |
| _ari_ | dmsimard: Hi kb gave us credentials to access the openstack instance in Centos do you know anything about that instance | 15:35 |
| *** rbuzatu has joined #openstack-infra | 15:35 | |
| clarkb | pabelanger: was the systemd thing related to the urandom fixer? I dont think we have touched systemd related things otherwise | 15:36 |
| *** tonytan4ever has quit IRC | 15:37 | |
| pabelanger | clarkb: we believe so, systemd was giving some warnings about permissions. That is what we fixed | 15:37 |
| *** rbuzatu has quit IRC | 15:38 | |
| pabelanger | clarkb: we had problems getting access to the console last night, so today I'm hoping things will be better locally | 15:38 |
| *** rbuzatu has joined #openstack-infra | 15:38 | |
| pabelanger | I've tested our changes manually, and they work on an existing ubuntu-xenial build. So, it is possible we are having a first boot issue some how | 15:38 |
| *** e0ne_ has quit IRC | 15:39 | |
| *** javeriak has joined #openstack-infra | 15:39 | |
| _ari_ | weshay: any idea why I would get this if I have the right credentials is it my client? | 15:39 |
| _ari_ | No handlers could be found for logger "keystoneauth.identity.generic.base" | 15:39 |
| _ari_ | ERROR (ConnectFailure): Unable to establish connection to http://controller.openstack.ci.centos.org:5000/v3/auth/tokens | 15:39 |
| weshay | _ari_, I don't know, maybe he's currently working on it | 15:40 |
| _ari_ | weshay: ok thanks | 15:40 |
| *** xarses has quit IRC | 15:41 | |
| openstackgerrit | Alex Schultz proposed openstack-infra/project-config: Update puppet integration jobs to use bindep.txt https://review.openstack.org/355203 | 15:41 |
| *** xarses has joined #openstack-infra | 15:41 | |
| openstackgerrit | Alex Schultz proposed openstack-infra/project-config: Update puppet integration jobs to use bindep.txt https://review.openstack.org/355203 | 15:42 |
| fungi | electrofelix: zxiiro: waynr: zaro: following our discussion, i've added kien ha to the jenkins-job-builder-core reviewer group in gerrit. do you happen to know a corresponding irc nick? i like to know how to raise core reviewers in irc when possible | 15:42 |
| *** _coolsvap_ is now known as coolsvap | 15:42 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Disentangle YamlParser and ModuleRegistry classes https://review.openstack.org/319622 | 15:43 |
| *** irtermite has joined #openstack-infra | 15:43 | |
| *** javeriak_ has joined #openstack-infra | 15:43 | |
| waynr | I don't, sorry | 15:43 |
| openstackgerrit | Alex Schultz proposed openstack-infra/project-config: Update puppet integration jobs to use bindep.txt https://review.openstack.org/355203 | 15:43 |
| * irtermite waves at cloudnull and d34dh0r53 *popcorn* | 15:44 | |
| * d34dh0r53 waves back | 15:44 | |
| cloudnull | ohai | 15:44 |
| *** javeriak has quit IRC | 15:44 | |
| irtermite | bad things cloudnull bad things | 15:45 |
| *** salv-orlando has joined #openstack-infra | 15:45 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Factor XmlJobGenerator out of YamlParser. https://review.openstack.org/319623 | 15:47 |
| electrofelix | fungi: pretty certain he uses kien-ha or kien_ha, I can't recall which one cause I always have autocomplete on nicks turned on | 15:47 |
| *** yamamoto has quit IRC | 15:47 | |
| fungi | thanks electrofelix | 15:47 |
| fungi | electrofelix: neither of those nicks seem to be active in freenode right now. any idea what his typical hours are and/or channels he frequents? | 15:48 |
| *** xarses has quit IRC | 15:49 | |
| *** xarses has joined #openstack-infra | 15:49 | |
| *** links has quit IRC | 15:49 | |
| fungi | electrofelix: from my irc client log, it seems there was a kien-ha in here ~5 days ago | 15:49 |
| fungi | so i guess that's it | 15:49 |
| electrofelix | fungi: that makes it easier I guess ;-) | 15:50 |
| *** jtomasek has quit IRC | 15:50 | |
| *** yamamoto has joined #openstack-infra | 15:50 | |
| fungi | only seems to pop into this channel for a few minutes up to a couple hours at a time though | 15:50 |
| electrofelix | glad to see I could remember the correct one as being the mostly likely version of it | 15:50 |
| electrofelix | fungi: I tend to suppress join/leave records for here because it's such a large channel, had assumed he's limited at the moment as he has an exam | 15:52 |
| electrofelix | but wouldn't have been sure about level of presence in here in general | 15:52 |
| *** annegentle has quit IRC | 15:54 | |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Fix Pagination for Admin Users Page https://review.openstack.org/354736 | 15:54 |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Prettify Task-Status-Changes in Recent Events https://review.openstack.org/357257 | 15:54 |
| fungi | electrofelix: good to know. i'll get up with him the next time i see him around | 15:55 |
| *** Ravikiran_K has joined #openstack-infra | 15:55 | |
| *** derekh has quit IRC | 15:56 | |
| openstackgerrit | Merged openstack-infra/storyboard: Describe Storyboard in more detail https://review.openstack.org/356021 | 15:56 |
| openstackgerrit | Ryan Tidwell proposed openstack-infra/project-config: Make gate-neutron-dynamic-routing-dsvm-functional voting. https://review.openstack.org/357980 | 15:57 |
| *** beagles is now known as beagles_brb | 15:59 | |
| *** yamamoto has quit IRC | 15:59 | |
| *** Ravikiran_K_ has joined #openstack-infra | 16:00 | |
| *** Ravikiran_K has quit IRC | 16:00 | |
| *** matrohon has quit IRC | 16:02 | |
| *** yamamoto has joined #openstack-infra | 16:02 | |
| *** shashank_hegde has joined #openstack-infra | 16:03 | |
| *** jaosorior has quit IRC | 16:04 | |
| *** xarses_ has joined #openstack-infra | 16:04 | |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Allow ramdisk-create to run without $USER set https://review.openstack.org/357985 | 16:05 |
| pabelanger | clarkb: any thought about changing our our volume on nodepool from SATA to SSD? | 16:06 |
| pabelanger | clarkb: in an effort to get more IO performance | 16:06 |
| fungi | pabelanger: i thought we tried that at one point but weren't actually heavily i/o bound anyway? | 16:07 |
| *** xarses has quit IRC | 16:07 | |
| pabelanger | fungi: Oh, not sure. I just know we take a while now to copy data around on the server | 16:07 |
| *** raunak has joined #openstack-infra | 16:08 | |
| pabelanger | pushing 40 mins to just a mv command | 16:08 |
| *** yamamoto has quit IRC | 16:08 | |
| clarkb | when I made the switch to the bigger volume the times were roughly the same | 16:08 |
| clarkb | 45 minutes to build an image | 16:08 |
| fungi | is it /dev/xvdb we're doing most of this on? | 16:08 |
| fungi | just looking at cacti graphs to see what things are looking like there | 16:09 |
| clarkb | the 1TB device | 16:09 |
| fungi | i'll check vgs | 16:09 |
| clarkb | dib builds in a chroot then has to copy all those contents into the actual image | 16:10 |
| fungi | er, pvs | 16:10 |
| fungi | and yeah, it's xvdb | 16:10 |
| clarkb | thats the slow mv and may go faster with ssd | 16:10 |
| pabelanger | ya | 16:10 |
| pabelanger | or just use a lot of RAM for tmpfs :) | 16:11 |
| *** sdague has quit IRC | 16:11 | |
| clarkb | you need at least 13GB right now I think | 16:12 |
| clarkb | :) | 16:12 |
| fungi | looks like we spike up around 13-14 mb/sec read and write, and around 2.8kiops read and write | 16:12 |
| fungi | honestly, the graphs show we're much more constrainted on ram, cpu and network bandwidth | 16:12 |
| fungi | looks like we could especially benefit from more ram | 16:13 |
| *** mat128 is now known as mat128|afk | 16:13 | |
| fungi | it's mostly active/resident, so very little left for filesystem caching | 16:13 |
| *** sebian has quit IRC | 16:14 | |
| pabelanger | I still want to discuss having nodepool.o.o and nodepool-builder.o.o to help with that, but I think we want to wait until zuulv3 for that | 16:14 |
| *** jianghuaw has quit IRC | 16:14 | |
| openstackgerrit | Thanh Ha proposed openstack-infra/jenkins-job-builder: Simplify delete by removing unnecessary loop https://review.openstack.org/357990 | 16:14 |
| fungi | though cpu iowait is looking pretty bad | 16:14 |
| zxiiro | fungi: Kien's irc nick is kien-ha | 16:14 |
| clarkb | pabelanger: we shouldnt need to wait for zuulv3 to do that | 16:14 |
| fungi | ssd might help some with iowait, but more cpu is probably in order anyway | 16:14 |
| clarkb | pabelanger: we'll just change whatever thr "message" bus is regardless | 16:15 |
| fungi | zxiiro: thanks! between electrofelix's memory and my irc client logs, i managed to figure it out | 16:15 |
| pabelanger | clarkb: maybe I missed understood the objections raise when I last talked about it | 16:15 |
| clarkb | pabelanger: but thats likely a reasonably next step along with going ssd | 16:15 |
| pabelanger | clarkb: I did upload a few patches this week to update puppet, let me find them | 16:16 |
| fungi | clarkb: pabelanger: yeah, launch a nodepool builder instance with more ram/cpu/bandwidth and appropriately sized ssd cinder volume (and we could probably scale down nodepool.o.o some after that) | 16:16 |
| pabelanger | https://review.openstack.org/#/c/356484/ is an easy one | 16:17 |
| *** matbu is now known as matbu|afk | 16:17 | |
| jeblair | i think a new builder host at any time would be fine; multiple even if we want. be sure to name the first nodepool-builder01. :) | 16:17 |
| fungi | is the builder architecture such that we can attach multiple builders and distribute the work between them? | 16:17 |
| *** ihrachys has quit IRC | 16:17 | |
| openstackgerrit | Darragh Bailey proposed openstack-infra/jenkins-job-builder: Ensure logging enabled for CLI https://review.openstack.org/357994 | 16:17 |
| fungi | if so, then we might not need to make it super beefy after all | 16:18 |
| jeblair | fungi: yes -- we just need slightly different nodepool.yaml files | 16:18 |
| *** john-davidge has quit IRC | 16:18 | |
| *** shashank_hegde has quit IRC | 16:18 | |
| jeblair | actually that may not even be necessary for us | 16:18 |
| *** sbadia has joined #openstack-infra | 16:18 | |
| fungi | the nodepool.o.o bw graphs top out at 400mbps daily, so i'm guessing that's our cap for the current flavor | 16:19 |
| fungi | our iowait spikes are also mostly around when we're maxing out the network, so it's more likely coming from that than disk i/o bandwidth and latency | 16:20 |
| *** coolsvap is now known as _coolsvap_ | 16:20 | |
| *** esikachev has quit IRC | 16:20 | |
| fungi | though lining up the graphs for network bw, xvdb iops and cpu usage it looks like a combination of both | 16:21 |
| *** matthewbodkin has quit IRC | 16:21 | |
| *** dprince has joined #openstack-infra | 16:21 | |
| clarkb | pabelanger: another thing I noticed is that when we claer the dib cache out we get slightly smaller images. So probably not worth saving the old cache, just let it rebuild | 16:21 |
| fungi | iowait is worse when we're slamming the network and the disk at the same time, not as terrible when we're just slamming the disk and not the network | 16:22 |
| fungi | which makes sense | 16:22 |
| *** mriedem is now known as mriedem_away | 16:22 | |
| pabelanger | clarkb: sure | 16:22 |
| fungi | but still bad even when just disk | 16:22 |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-nodepool: Remove nodepool-builder from nodepool::init https://review.openstack.org/356498 | 16:22 |
| *** amotoki_ has quit IRC | 16:22 | |
| fungi | i have a feeling more ram to give us better disk caching would relieve a bit of this | 16:23 |
| openstackgerrit | Ryan Tidwell proposed openstack-infra/project-config: Make gate-neutron-dynamic-routing-dsvm-functional voting. https://review.openstack.org/357980 | 16:24 |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-openstackci: Use nodepool::builder for nodepool https://review.openstack.org/356503 | 16:24 |
| pabelanger | fungi: clarkb: okay, that stack should get up started by breaking out nodepool-builder class | 16:24 |
| pabelanger | which we need to land first | 16:24 |
| *** jordanP has quit IRC | 16:25 | |
| *** tonytan_brb has quit IRC | 16:25 | |
| clarkb | pabelanger: I will start reviewing them now. | 16:25 |
| fungi | yeah, getting our image building and upload to complete sooner in the day would be extra nice | 16:25 |
| fungi | i will try to get some time to go through them later today | 16:25 |
| fungi | thanks pabelanger! | 16:26 |
| pabelanger | np | 16:26 |
| pabelanger | it will take a few iterations of the patch to break them up while not breaking production | 16:26 |
| *** tonytan4ever has joined #openstack-infra | 16:26 | |
| *** rbuzatu has quit IRC | 16:26 | |
| *** jpich has quit IRC | 16:28 | |
| *** asselin_ has joined #openstack-infra | 16:28 | |
| clarkb | pabelanger: you will also need the nodepool user, the vcsrepo, etc on the builder | 16:29 |
| clarkb | pabelanger: might make sense to keep all that common stuff in init.pp then add in a server.pp and builder.pp | 16:29 |
| pabelanger | clarkb: Ya, I haven't done that step yet. But seems like the right approach | 16:29 |
| *** mfedosin has quit IRC | 16:29 | |
| clarkb | pabelanger: in https://review.openstack.org/#/c/356484/1 should you remove the diskimage_builder include in init.pp? | 16:30 |
| *** ddieterly has joined #openstack-infra | 16:31 | |
| *** asselin__ has quit IRC | 16:31 | |
| pabelanger | clarkb: I thought about that, but some reason didn't. | 16:31 |
| pabelanger | I guess we should since diskimage-builder is only a dependency for nodepool-builder now | 16:32 |
| *** esberglu has quit IRC | 16:32 | |
| clarkb | pabelanger: ya I think it will make the code cleaner and avoid confusion over where dib is needed | 16:32 |
| clarkb | cloudnull: did you see https://review.openstack.org/#/c/357764/1? | 16:33 |
| clarkb | rcarrillocruz: sdague ^ there is also a comment in there that should likely be addressed | 16:33 |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-nodepool: Remove nodepool-builder from nodepool::init https://review.openstack.org/356498 | 16:33 |
| pabelanger | oops | 16:34 |
| pabelanger | bad rebase | 16:34 |
| cloudnull | clarkb: yea i caught that cidr too | 16:34 |
| fungi | clarkb: pabelanger: just remember to keep the default deployment backward-compatible with an all-in-one (or at least land corresponding changes in puppet-openstackci) | 16:35 |
| cloudnull | asked, sdague about it earlier but decided to just roll it in because the PS had +w. | 16:35 |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-nodepool: Remove nodepool-builder from nodepool::init https://review.openstack.org/356498 | 16:35 |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-nodepool: include ::diskimage_builder in builder.pp too https://review.openstack.org/356484 | 16:35 |
| fungi | we have downstream users of the nodepool module who would be surprised if their server suddenly stopped being a builder | 16:35 |
| pabelanger | fungi: Ya, that's what I am trying to do | 16:35 |
| pabelanger | fungi: as long as the entry point is puppet-openstackci, they are fine | 16:36 |
| pabelanger | puppet-nodepool we might been some breaking changes | 16:36 |
| pabelanger | however, I can send out an email to ML about that | 16:36 |
| fungi | that would probably be good regardless, yeah | 16:36 |
| openstackgerrit | Wayne Warren proposed openstack-infra/jenkins-job-builder: Be explicit about objects exported by modules. https://review.openstack.org/319625 | 16:37 |
| clarkb | the changes that are currently up are backward compatible | 16:37 |
| fungi | cool, figured they probably were | 16:37 |
| clarkb | since builder is already split out and only it needs dib | 16:37 |
| krotscheck | mordred (or other): Keystone asks for a project_id for an explicitly scoped auth call; In clouds.yaml, is that the 'project_domain_id', the 'project_name', or the 'user_domain_id'? | 16:37 |
| krotscheck | Or both? | 16:38 |
| krotscheck | Or all three? | 16:38 |
| clarkb | pabelanger: if we do init.pp server.pp and builder.pp we can have init.pp include server.pp with a toggleable flag to maintain backward compat there too. Then we would flip the toggle to avoid installing the server on the builder node | 16:38 |
| *** yamahata has quit IRC | 16:38 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Improve debug output from tests https://review.openstack.org/358008 | 16:38 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Combine shared queues explicitly https://review.openstack.org/358009 | 16:38 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Re-enable the shared/independent queue test https://review.openstack.org/358010 | 16:38 |
| clarkb | krotscheck: its project_name. projects and users go in domains but they have their own ids | 16:39 |
| fungi | clarkb: pabelanger: that's roughly what i was envisioning to avoid downstream surprises | 16:39 |
| clarkb | krotscheck: well use project_id if you have the uuid and project_name if you have the human readable name | 16:39 |
| pabelanger | fungi: clarkb: yes, I am on board with that | 16:40 |
| krotscheck | clarkb: Hrm. That's... a lot of edge cases. | 16:40 |
| fungi | krotscheck: that's openstack :/ | 16:40 |
| clarkb | you should also provide the domain info if using keystone v3 | 16:41 |
| fungi | why have one unique id when you can have many? | 16:41 |
| * krotscheck nukes openstack from orbit | 16:41 | |
| clarkb | fungi: well because someone decided that you should be able to have non unique ids that are unique! | 16:41 |
| fungi | yep | 16:41 |
| *** mfedosin has joined #openstack-infra | 16:41 | |
| krotscheck | I'm reading a little frustration here ;) | 16:42 |
| * krotscheck wonders if the keystone devs have actually used keystone. | 16:42 | |
| fungi | it's fallout from the way keystone federation and domains work | 16:42 |
| fungi | there are reasons it's done that way, but they're non-obvious | 16:42 |
| clarkb | basically I don't get to be the only clarkb in the world anymore | 16:42 |
| clarkb | you could be a clarkb too | 16:42 |
| clarkb | we just have to be in different domains | 16:42 |
| *** krotscheck is now known as the_other_clarkb | 16:42 | |
| the_other_clarkb | There we go | 16:42 |
| fungi | there can be only one^H^H^Htwo! | 16:43 |
| * the_other_clarkb is the clarkb of the javascript domain | 16:43 | |
| clarkb | and we can work on projects that share names but are otherwise distinct | 16:43 |
| * the_other_clarkb unfortunately cannot talk to the first clarkb, because we're in different domains | 16:43 | |
| *** the_other_clarkb is now known as krotscheck | 16:43 | |
| clarkb | say if javascript and python both have yaml parser projects we can both call them yaml | 16:43 |
| * krotscheck sighs | 16:43 | |
| fungi | and if you ever met in person, the universe would implode | 16:44 |
| krotscheck | Ok. So, I have a clouds.yaml, which may-or-may-not contain a project_domain_id, a user_domain_id, and a project_name. How in the wordl do I construct that body? | 16:44 |
| clarkb | pabelanger: for https://review.openstack.org/#/c/356498/4/manifests/init.pp we probably want to do a similar toggle flag like I mentioend there too :/ the code gets ugly but remains backward compat | 16:44 |
| clarkb | pabelanger: basically have a configure_builder = true, default and make it falseable | 16:45 |
| clarkb | otherwise we do have the puppet-openstackci issues fungi was talking about | 16:45 |
| *** Goneri has joined #openstack-infra | 16:45 | |
| *** rbuzatu has joined #openstack-infra | 16:46 | |
| *** Apoorva has joined #openstack-infra | 16:46 | |
| clarkb | krotscheck: let me find a link to our files | 16:46 |
| *** ddieterly is now known as ddieterly[away] | 16:46 | |
| clarkb | krotscheck: https://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/templates/nodepool/clouds.yaml.erb#n65 should illustrate what it looks like in clouds.yaml | 16:47 |
| krotscheck | clarkb: Right, I got that from the devstack-generated clouds.yaml | 16:47 |
| *** rbuzatu has quit IRC | 16:47 | |
| *** rbuzatu has joined #openstack-infra | 16:48 | |
| *** Ravikiran_K_ has quit IRC | 16:48 | |
| clarkb | pabelanger: left that in a proper review comment | 16:50 |
| *** roxanaghe has joined #openstack-infra | 16:51 | |
| *** annegentle has joined #openstack-infra | 16:52 | |
| *** beagles_brb is now known as beagles | 16:52 | |
| *** tesseract- has quit IRC | 16:53 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-nodepool: Refactor code from init.pp to server.pp https://review.openstack.org/358014 | 16:53 |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-nodepool: Add back nodepool::builder to init.pp https://review.openstack.org/358015 | 16:53 |
| clarkb | krotscheck: if you want to know how you find out what your domain, user, and project are I believe that horizon has a page that will tell you | 16:53 |
| pabelanger | clarkb: looking | 16:53 |
| pabelanger | also, just pushed up code based on what we just discussed | 16:54 |
| openstackgerrit | Wayne Warren proposed openstack-infra/jenkins-job-builder: Cleanup various deprecation warnings. https://review.openstack.org/319624 | 16:55 |
| clarkb | pabelanger: looks like the xenial build just finished | 16:55 |
| *** amotoki has joined #openstack-infra | 16:55 | |
| pabelanger | cool | 16:55 |
| pabelanger | let me try launching it in my test environment | 16:56 |
| *** julim has quit IRC | 16:56 | |
| clarkb | thankfully osic image uploads are pretty quick | 16:56 |
| *** matt-borland has quit IRC | 16:56 | |
| *** camunoz has quit IRC | 16:56 | |
| *** ddieterly[away] is now known as ddieterly | 16:57 | |
| *** tosky has quit IRC | 16:57 | |
| *** rbuzatu has quit IRC | 16:59 | |
| *** rbuzatu has joined #openstack-infra | 17:02 | |
| cloudnull | pabelanger: fingers crossed :) | 17:04 |
| *** ociuhandu has quit IRC | 17:05 | |
| pabelanger | Waiting up upload to finish, then will manually test it | 17:05 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: WIP: Rework the task list layout https://review.openstack.org/357306 | 17:06 |
| *** sdague has joined #openstack-infra | 17:07 | |
| openstackgerrit | Darragh Bailey proposed openstack-infra/jenkins-job-builder: Support explicit API and simple config creation https://review.openstack.org/358019 | 17:07 |
| *** roxanaghe has quit IRC | 17:07 | |
| *** mhickey has quit IRC | 17:08 | |
| *** pvaneck has joined #openstack-infra | 17:09 | |
| fungi | need to go do battle with my yard before the rains return. bbiab | 17:09 |
| *** tphummel has joined #openstack-infra | 17:10 | |
| *** ddieterly is now known as ddieterly[away] | 17:10 | |
| *** yamahata has joined #openstack-infra | 17:12 | |
| clarkb | pabelanger: I have reviewed the puppet-nodepool stack | 17:13 |
| mordred | krotscheck: project_id . project_domain_id is how you find the domain that a project is in, and is important if you need to find a project by name | 17:13 |
| krotscheck | mordred: Thanks. clarkb helped out a lot. | 17:13 |
| mordred | cool | 17:13 |
| mordred | krotscheck: btw - don't know if you're all the way to this or not - but these answers are only valid if the keystone in question uses the Password auth plugin | 17:14 |
| *** roxanaghe has joined #openstack-infra | 17:15 | |
| mordred | other auth plugins may or may not need parameters such as project_id or user{name,id} | 17:15 |
| dmsimard | _ari_: hey, do you want to come chat about it on #centos-devel? Not sure it's relevant here. | 17:15 |
| krotscheck | mordred: Well, I welcome those persons to contribute their own configuration to the JavaScript SDK :) | 17:15 |
| mordred | krotscheck: :) | 17:15 |
| mordred | krotscheck: I mostly just mention it so that you don't assume those are always-existing hard-coded parameters and/or leave the space for someone to make the module that does that | 17:16 |
| mordred | krotscheck: I do't think there is much pressing need for you to implement anyhting other than password auth for the forseeable future | 17:16 |
| *** lucasagomes is now known as lucas-afk | 17:18 | |
| *** mhickey has joined #openstack-infra | 17:19 | |
| krotscheck | mordred: At this point, anything other than our authentication methods require an externally provided token, because we don't want to presume your token caching strategy. | 17:23 |
| krotscheck | So, yeah- lots of opportunity to extend/add methods. | 17:23 |
| *** electrofelix has quit IRC | 17:24 | |
| *** kaisers1 has quit IRC | 17:25 | |
| *** shashank_hegde has joined #openstack-infra | 17:25 | |
| *** sambetts is now known as sambetts|afk | 17:28 | |
| *** kaisers has joined #openstack-infra | 17:29 | |
| *** baoli has quit IRC | 17:29 | |
| *** kaisers1 has joined #openstack-infra | 17:29 | |
| *** baoli has joined #openstack-infra | 17:30 | |
| *** stewie925 has quit IRC | 17:30 | |
| openstackgerrit | Jay Faulkner proposed openstack-infra/project-config: Add docs jobs to ironic-lib https://review.openstack.org/356797 | 17:30 |
| clarkb | pabelanger: any luck booting xenial? | 17:31 |
| pabelanger | clarkb: nope | 17:31 |
| pabelanger | I don't think SSH is running | 17:31 |
| pabelanger | let me get a log | 17:31 |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Rename "parallelize" decorator to "concurrent" https://review.openstack.org/333780 | 17:32 |
| openstackgerrit | Jay Faulkner proposed openstack-infra/project-config: Add docs jobs to ironic-lib https://review.openstack.org/356797 | 17:32 |
| pabelanger | clarkb: http://paste.openstack.org/show/561446/ | 17:32 |
| pabelanger | I don't see ssh keys for server getting generated | 17:32 |
| *** kzaitsev_mb has quit IRC | 17:33 | |
| *** rbrndt has quit IRC | 17:34 | |
| pabelanger | it must be our systemd file | 17:34 |
| clarkb | pabelanger: and did that build with the new urandom seed unit file? | 17:34 |
| pabelanger | yes | 17:34 |
| pabelanger | you can see it starting in the log | 17:34 |
| pabelanger | along with unbound starting properly | 17:34 |
| pabelanger | we run it before pre-network.target | 17:35 |
| pabelanger | I wonder if that is causing problems | 17:35 |
| _ari_ | dmsimard: sure | 17:35 |
| jeblair | pabelanger: this worked on the test host, right? | 17:35 |
| pabelanger | jeblair: yes | 17:35 |
| pabelanger | that is the frustration part | 17:36 |
| pabelanger | if I manually do this on an image, it works | 17:36 |
| jeblair | pabelanger: so maybe it's the ssh key generation itself? maybe we should see if we can trigger that on the test host? | 17:36 |
| jeblair | (though -- that's probably a one-way trip :) | 17:36 |
| *** dtantsur is now known as dtantsur|afk | 17:36 | |
| pabelanger | jeblair: ya, we could do that | 17:36 |
| pabelanger | see what happens | 17:36 |
| *** Na3iL has quit IRC | 17:37 | |
| clarkb | I do notice that the starting Load/Save random seed happens after the 1 byte urandom raed | 17:37 |
| openstackgerrit | Merged openstack-infra/zuul: Stub out aborting jobs in ansible launcher https://review.openstack.org/353778 | 17:37 |
| clarkb | oh that may be related to haveged starting for real | 17:37 |
| jeblair | clarkb: it says it's systemd-udevd reading it | 17:38 |
| openstackgerrit | Merged openstack-infra/zuul: Re-enable test_failed_changes https://review.openstack.org/354419 | 17:38 |
| jeblair | (that's a kernel message that means "this process read from urandom before it was initialized") | 17:39 |
| openstackgerrit | Merged openstack-infra/puppet-mediawiki: image_scaler: Partially separate packages for Trusty https://review.openstack.org/352056 | 17:39 |
| jeblair | hopefully systemd isn't using that for anything important :) | 17:39 |
| *** amoralej is now known as amoralej|pto | 17:39 | |
| jeblair | but it was there before we started all this | 17:39 |
| clarkb | where in the log is jeblair's script recorded? | 17:39 |
| *** baoli has quit IRC | 17:40 | |
| jeblair | [[0;32m OK [0m] Started Quickly initialize the nonb...andom number generator at boot.. | 17:40 |
| jeblair | that one? | 17:40 |
| *** esberglu has joined #openstack-infra | 17:40 | |
| openstackgerrit | Merged openstack-infra/puppet-mediawiki: Also require php5-memcached https://review.openstack.org/352057 | 17:41 |
| clarkb | aha! I can't grep for "random" because it truncated the log | 17:41 |
| jeblair | [[0;1;31m SKIP [0m] Ordering cycle found, skipping Network (Pre) | 17:41 |
| jeblair | [[0;1;31m SKIP [0m] Ordering cycle found, skipping Network | 17:41 |
| clarkb | or squiched its ends together | 17:41 |
| jeblair | that looks important | 17:41 |
| jeblair | though it looks like it does do it later | 17:42 |
| jeblair | [[0;32m OK [0m] Reached target Network. | 17:42 |
| *** mriedem_away is now known as mriedem | 17:43 | |
| *** mhickey has quit IRC | 17:44 | |
| jeblair | [[0;32m OK [0m] Started OpenBSD Secure Shell server. | 17:45 |
| jeblair | it also did that | 17:45 |
| pabelanger | oh, I messed that | 17:46 |
| pabelanger | so SSH should be running | 17:46 |
| *** baoli has joined #openstack-infra | 17:46 | |
| *** yamamoto has joined #openstack-infra | 17:47 | |
| pabelanger | I don't see glean actually | 17:49 |
| *** mfedosin has quit IRC | 17:49 | |
| pabelanger | let me make sure I enable config-drive | 17:49 |
| *** julim has joined #openstack-infra | 17:50 | |
| pabelanger | yes, is enabled | 17:51 |
| clarkb | is it possible glean isn't running beacuse it runs as part of network-pre which was identified as part of the loop and is somehow getting axed? | 17:51 |
| * clarkb looks at glean's unit file | 17:51 | |
| openstackgerrit | Andrea Scarpino proposed openstack-infra/jenkins-job-builder: Reporters: support for perModuleEmail field https://review.openstack.org/352799 | 17:52 |
| *** yamamoto has quit IRC | 17:52 | |
| *** annegentle has quit IRC | 17:53 | |
| clarkb | https://git.openstack.org/cgit/openstack-infra/glean/tree/glean/init/glean@.service | 17:53 |
| openstackgerrit | Merged openstack-infra/puppet-mediawiki: Ensure cache directory is owned by the right user/group https://review.openstack.org/352058 | 17:53 |
| pabelanger | clarkb: jeblair: this is a working ubuntu-xenail console log: http://paste.openstack.org/show/561448/ | 17:54 |
| pabelanger | with out our urandom.service file | 17:54 |
| clarkb | pabelanger: the glean unit looks mostly sane except the ordering is network -> glean -> network-pre which is a little funny (why is network-pre after network?) | 17:55 |
| *** e0ne has joined #openstack-infra | 17:55 | |
| pabelanger | Ya, I see that | 17:55 |
| pabelanger | not sure why that is | 17:55 |
| clarkb | pabelanger: the same ordering cycle shows up in both though | 17:55 |
| *** baoli has quit IRC | 17:56 | |
| *** baoli has joined #openstack-infra | 17:56 | |
| clarkb | pabelanger: is it possible that the dib-init-system element could be breaking for simple-init? | 17:58 |
| sdague | ok... so anyone want to help us move forward on the drop pg jobs plan - which requires this d-g change to not drop metadata server testing - https://review.openstack.org/#/c/357443/ | 17:58 |
| clarkb | pabelanger: like maybe booting is working "fine" its our image build that is unhappy | 17:58 |
| *** inc0 has quit IRC | 17:58 | |
| pabelanger | clarkb: Ya, we've never used it before. in fact, I am considering removing it and switching to puppet | 17:59 |
| clarkb | according to the README for that element we are using it correctly | 18:00 |
| *** psachin has joined #openstack-infra | 18:01 | |
| *** tqtran has joined #openstack-infra | 18:01 | |
| pabelanger | 1 sec | 18:02 |
| pabelanger | I think I found something | 18:02 |
| pabelanger | it does look like glean is not starting on my test instance now | 18:05 |
| *** tonytan4ever has quit IRC | 18:06 | |
| *** tonytan4ever has joined #openstack-infra | 18:06 | |
| * clarkb tries a local build without all of the big content | 18:07 | |
| *** Sukhdev has joined #openstack-infra | 18:08 | |
| pabelanger | http://paste.openstack.org/show/561451/ | 18:08 |
| pabelanger | that is the error I am seeing when I try to manually start glean server | 18:08 |
| pabelanger | service* | 18:08 |
| *** tonytan4ever has quit IRC | 18:08 | |
| *** yaume has joined #openstack-infra | 18:09 | |
| pabelanger | I wonder if I am starting it wrong | 18:09 |
| openstackgerrit | Paul Van Eck proposed openstack-infra/puppet-refstack: Make sure libffi-dev is installed https://review.openstack.org/358034 | 18:09 |
| clarkb | pabelanger: I believe that the addr_assign_type is figuring out what kind of mac address you have | 18:10 |
| clarkb | pabelanger: how did you run it? it almost looks like the iface arg is not right | 18:10 |
| pabelanger | sudo systemctl start glean\@.service.service | 18:11 |
| *** Goneri has quit IRC | 18:11 | |
| clarkb | ya I think one of the .service's needs to be the interface name | 18:11 |
| clarkb | based on my reading of that traceback | 18:11 |
| *** tosky has joined #openstack-infra | 18:11 | |
| pabelanger | that is better | 18:12 |
| pabelanger | okay, however with urandom enabled, glean no longer runs on boot | 18:13 |
| pabelanger | let me see if I can fix that | 18:13 |
| clarkb | my local build with initialize-urandom is building | 18:14 |
| clarkb | with a very minimal elements list which should at least rule out something happening in puppet | 18:15 |
| pabelanger | so when should we be running glean? | 18:15 |
| pabelanger | before networking is setup or after? | 18:15 |
| pabelanger | or both | 18:15 |
| pabelanger | maybe that is why it has both before and after | 18:15 |
| *** amotoki has quit IRC | 18:16 | |
| clarkb | glean happens before I think since its job is to write out the netwrok interface specificatio nfiles | 18:16 |
| *** tonytan4ever has joined #openstack-infra | 18:16 | |
| *** dtardivel has quit IRC | 18:17 | |
| clarkb | pabelanger: I do notice that the simple-init element seems to not explicitly enable the service like we have in initialize-urandom | 18:17 |
| *** yaume has quit IRC | 18:17 | |
| pabelanger | clarkb: right, but growroot does | 18:17 |
| pabelanger | which I based the change on | 18:17 |
| clarkb | ah ok. Was thinking maybe simple-init doesn't do enough in all cases to be enabled on boot | 18:18 |
| *** sadmordred has quit IRC | 18:18 | |
| *** armax has quit IRC | 18:18 | |
| pabelanger | clarkb: so, I moved initialize-urandom.server to After=network.target | 18:18 |
| pabelanger | glean started | 18:18 |
| pabelanger | and dns worked | 18:18 |
| clarkb | huh | 18:19 |
| pabelanger | http://paste.openstack.org/show/561452/ | 18:19 |
| clarkb | which js what glean does that looks funny | 18:19 |
| pabelanger | err | 18:19 |
| pabelanger | to large I guess | 18:19 |
| pabelanger | http://pastebin.com/raw/TGcfRvEM | 18:20 |
| *** rbrndt has joined #openstack-infra | 18:20 | |
| *** baoli has quit IRC | 18:21 | |
| clarkb | pabelanger: this was on the newly built image too? | 18:21 |
| pabelanger | this was old image, where I manually installed urandom.server | 18:21 |
| pabelanger | service* | 18:22 |
| pabelanger | but with After=network.target | 18:22 |
| pabelanger | and glean is happier for some reason | 18:22 |
| *** armax has joined #openstack-infra | 18:23 | |
| *** e0ne has quit IRC | 18:23 | |
| pabelanger | with after=network.target: http://paste.openstack.org/show/561453/ | 18:24 |
| pabelanger | before=network-pre.target: http://paste.openstack.org/show/561454/ | 18:25 |
| clarkb | pabelanger: did you do both the befire and after like glean does or just one or the other? | 18:25 |
| *** baoli has joined #openstack-infra | 18:26 | |
| pabelanger | clarkb: for urandom we only have before and want | 18:26 |
| pabelanger | let me make both the same | 18:26 |
| pabelanger | I'll update urandom to match glean | 18:26 |
| *** ddieterly[away] is now known as ddieterly | 18:27 | |
| *** Hal1 has joined #openstack-infra | 18:28 | |
| *** mlekkas has quit IRC | 18:28 | |
| *** erlon has joined #openstack-infra | 18:28 | |
| pabelanger | clarkb: that worked too | 18:28 |
| clarkb | so maybe we go with that? I think its ok for it to run at the same.time as glean since they have similar requiremnts | 18:29 |
| clarkb | and dont dep on each other | 18:29 |
| pabelanger | sure | 18:29 |
| *** tqtran has quit IRC | 18:29 | |
| *** tonytan4ever has quit IRC | 18:30 | |
| *** Hal has quit IRC | 18:30 | |
| *** tonytan4ever has joined #openstack-infra | 18:31 | |
| *** _ari_ has quit IRC | 18:32 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Add After=network.target for urandom.service https://review.openstack.org/358039 | 18:32 |
| *** yamamoto has joined #openstack-infra | 18:32 | |
| pabelanger | clarkb: fungi: That is our fix for networking on xenail, I hope | 18:32 |
| *** tqtran has joined #openstack-infra | 18:33 | |
| *** baoli has quit IRC | 18:33 | |
| fungi | awesome | 18:33 |
| *** armax has quit IRC | 18:34 | |
| fungi | reviewing now | 18:34 |
| clarkb | pabelanger: I +2'd and will attempt to test using my loxal builds as soon as I eat this blt | 18:34 |
| pabelanger | yes, I need me some food too | 18:34 |
| fungi | i guess i returned from groundskeeping duty just at the right time ;) | 18:35 |
| *** baoli has joined #openstack-infra | 18:36 | |
| fungi | i wonder if dib should have a "validate systemd" element or similar that checks for dependency loops and errors out at image build time? | 18:36 |
| *** yamamoto has quit IRC | 18:38 | |
| pabelanger | fungi: could be useful | 18:41 |
| *** sambetts|afk has quit IRC | 18:42 | |
| *** sambetts has joined #openstack-infra | 18:44 | |
| pabelanger | fungi: clarkb: I'm going to enqueue that in to gate so we can start our build sooner | 18:45 |
| clarkb | ok | 18:45 |
| fungi | sounds good, thanks | 18:45 |
| *** javeriak_ has quit IRC | 18:46 | |
| *** amotoki has joined #openstack-infra | 18:47 | |
| *** amotoki has quit IRC | 18:50 | |
| pvaneck | hey fungi, refstack recently updated the version of pyopenssl and this requires libffi-dev to be installed on the server in order to build correctly. | 18:51 |
| pvaneck | I submitted https://review.openstack.org/#/c/358034 , but it would be great if you could manually install the package on the refstack server. | 18:51 |
| pvaneck | seems likely that this is causing the refstack node to go unreported for a long time: http://puppetboard.openstack.org/node/refstack.openstack.org | 18:53 |
| fungi | pvaneck: which sdist failed to build? | 18:53 |
| fungi | or i guess i can go spelunking in the puppet logs | 18:53 |
| fungi | Aug 19 18:27:29 refstack puppet-user[21192]: (/Stage[main]/Refstack::App/Exec[install-refstack]/returns) c/_cffi_backend.c:15:17: fatal error: ffi.h: No such file or directory | 18:54 |
| *** valderrv has quit IRC | 18:54 | |
| pvaneck | yea, cffi failed to build | 18:54 |
| fungi | needed by cryptography | 18:54 |
| fungi | just wondering if i should roll back one of the git repos by one commit so that puppet will advance it forward, but it looks like it's retrying on every run so this will probably self-correct | 18:55 |
| fungi | anyway, it's installed now and the config management change approved | 18:56 |
| pvaneck | awesome, thanks! | 18:56 |
| fungi | so hopefully in a few minutes it'll update successfully but i'll try to remember to check back in on it | 18:56 |
| pvaneck | yea, I will keep an eye on it | 18:56 |
| *** amotoki has joined #openstack-infra | 18:56 | |
| *** aorourke has quit IRC | 18:58 | |
| clarkb | ok I have a local dib build going with that patch in place | 18:59 |
| *** tqtran has quit IRC | 18:59 | |
| *** valderrv_ has joined #openstack-infra | 19:00 | |
| *** annegentle has joined #openstack-infra | 19:01 | |
| *** markusry has quit IRC | 19:02 | |
| *** markusry has joined #openstack-infra | 19:02 | |
| *** burgerk has joined #openstack-infra | 19:04 | |
| *** amotoki has quit IRC | 19:04 | |
| *** Goneri has joined #openstack-infra | 19:04 | |
| *** jed56 has quit IRC | 19:05 | |
| clarkb | except I ran out of disk again... /me has restarted it now with more disk | 19:05 |
| *** markusry has quit IRC | 19:06 | |
| *** fifieldt has quit IRC | 19:07 | |
| sdague | so... what's up with the 200 nodes in building? | 19:12 |
| clarkb | sdague: thats the broken systemd on xenial | 19:12 |
| sdague | ah | 19:13 |
| sdague | so this is why things were silent crashing before? | 19:13 |
| *** tqtran has joined #openstack-infra | 19:13 | |
| *** salv-orl_ has joined #openstack-infra | 19:15 | |
| pabelanger | not crash, ansible-playbook losses network connection, and zuul will requeue | 19:15 |
| clarkb | no, the silent crashing we think was related to ipv6 privacy extensions. We also fixed urandom being slow. The new image with both these fixes has unworking ssh due to the urandom unit file (we think) | 19:15 |
| *** psilvad has quit IRC | 19:15 | |
| clarkb | so by fixing urandom fix we should get working ssh and no more unhappy ipv6 | 19:15 |
| pabelanger | ++ | 19:15 |
| pabelanger | Oh, I didn't read what sdague said properly | 19:17 |
| pabelanger | nodepool ready-script is failing to resolve DNS, because of urandom | 19:17 |
| *** salv-orlando has quit IRC | 19:17 | |
| pabelanger | so nodepoool is churning a lot of ubuntu-xenial nodes in osic-cloud1 | 19:17 |
| pabelanger | I've see the problem in ovh too, but not as bad | 19:17 |
| *** yamamoto has joined #openstack-infra | 19:18 | |
| clarkb | pabelanger: oh you deleted the previous xenial uplaod that didn't have working ssh? | 19:18 |
| pabelanger | clarkb: yes, last night | 19:18 |
| clarkb | I guess the effect is similar. In any case we *should* have this sorted out soonish | 19:18 |
| mordred | clarkb, pabelanger: still trying to grok scrollback | 19:18 |
| *** kzaitsev_mb has joined #openstack-infra | 19:19 | |
| irtermite | pabelanger: how many instances are running right now? | 19:19 |
| *** fifieldt has joined #openstack-infra | 19:19 | |
| irtermite | in osic-cloud1 | 19:19 |
| jroll | anyone noticing gerrit probs? | 19:19 |
| pabelanger | http://grafana.openstack.org/dashboard/db/nodepool-osic | 19:19 |
| jroll | sorry, problems* | 19:19 |
| irtermite | oops, forgot about that link | 19:19 |
| pabelanger | irtermite: not many right now | 19:19 |
| mordred | clarkb, pabelanger: I thought I was going to have useful things to say about glean - but I don't think Ido | 19:19 |
| irtermite | ah... pabelanger the change to 512 hasn't been pushed yet? limit still at 256? | 19:19 |
| TheJulia | jroll: I suspect it has decided to call it a friday :) | 19:19 |
| pabelanger | irtermite: ya, we need new ubuntu-xenial images first | 19:20 |
| irtermite | right on | 19:20 |
| pabelanger | which should be coming in the next 2 hours | 19:20 |
| irtermite | nice | 19:20 |
| *** psilvad has joined #openstack-infra | 19:20 | |
| *** valderrv_ has quit IRC | 19:20 | |
| irtermite | pabelanger: I really want to take a cluster-wide poll of the usage when we hit 512 | 19:20 |
| *** kgiusti has left #openstack-infra | 19:21 | |
| clarkb | jroll: no, what kind of problems? | 19:22 |
| *** yamamoto has quit IRC | 19:22 | |
| jroll | clarkb: a review loading sat for about a minute before I posted here and let myself get distracted | 19:23 |
| clarkb | pabelanger: ok I forgot to install devuser BUT I acn hit the ssh daemon its up and listening on 22 using your most recent patch so I think that will get us sorted | 19:23 |
| jroll | it did load eventually | 19:23 |
| pabelanger | clarkb: great | 19:23 |
| jroll | clarkb: seems to be okay now | 19:23 |
| clarkb | there was a gerrit GC spike about 9 hours ago | 19:23 |
| jroll | other had people in ironic too | 19:23 |
| clarkb | but the graph ahs been quiet since then | 19:23 |
| jroll | hrm, maybe quick network blip or something | 19:24 |
| clarkb | looks like we may have restarted it early today beacuse the memory usage falls way off right around that spike too | 19:24 |
| clarkb | jroll: ya I know hogepodge had ipv6 issues to rax recently | 19:24 |
| clarkb | it turns out that the comcast to rackspace path is really unreliable from the PNW | 19:24 |
| jroll | clarkb: no v6 here, unfortunately, and this is comcast in the midwest | 19:25 |
| irtermite | clarkb: does comcast not have IPv6 along the path somewhere? | 19:25 |
| irtermite | ah | 19:25 |
| clarkb | irtermite: no they just have some bad bit buckets along the way | 19:25 |
| irtermite | jroll: that might explain it ^^ cloudnull | 19:25 |
| clarkb | irtermite: seattle and denver regularly drop packets on the floor | 19:25 |
| irtermite | =( | 19:25 |
| *** salv-orl_ has quit IRC | 19:26 | |
| jroll | clarkb: I thought we used gerrit to avoid bitbucket | 19:26 |
| jroll | :D | 19:26 |
| cloudnull | huh? | 19:26 |
| pabelanger | clarkb: Hmm, it looks like we are launching the broken ubuntu-xenial image in osic-cloud1. I think we should remove ubuntu-xenail from osic-cloud1 in nodepool.yaml until we land our new image | 19:26 |
| *** sarob has joined #openstack-infra | 19:26 | |
| *** ddieterly is now known as ddieterly[away] | 19:26 | |
| jroll | irtermite: yeah, unsure what you're saying might explain it | 19:26 |
| clarkb | pabelanger: sounds like a reasonable thing to do | 19:26 |
| pabelanger | clarkb: going to put nodepool.o.o into emergency file for now, and manually do it. until we land our patch | 19:27 |
| jroll | all I know is people in pdx, east coast, midwest, and brazil saw a blip at the same time in ironic channel. it's back now, no big deal :) | 19:27 |
| clarkb | pabelanger: ok | 19:27 |
| *** salv-orlando has joined #openstack-infra | 19:27 | |
| *** asettle has joined #openstack-infra | 19:27 | |
| *** sarob has quit IRC | 19:27 | |
| *** spzala has joined #openstack-infra | 19:28 | |
| *** raunak has quit IRC | 19:28 | |
| clarkb | of course now virsh can't shutdown that instance and I can't ssh into it :| but hey ssh is running | 19:28 |
| *** sarob has joined #openstack-infra | 19:28 | |
| pabelanger | #status log nodepool.o.o added to emergency file on puppetmaster.o.o. So we can remove ubuntu-xenail label from osic-cloud1 | 19:28 |
| openstackstatus | pabelanger: finished logging | 19:28 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-mediawiki: Drop Ubuntu Precise compatibility https://review.openstack.org/358084 | 19:29 |
| irtermite | I retract that statement jroll | 19:29 |
| clarkb | pabelanger: your change has passed testing, just waiting on zuul/gerrit to merge it | 19:29 |
| clarkb | the event queue is rather large | 19:29 |
| jroll | irtermite: heh, no worries | 19:29 |
| pabelanger | clarkb: ya, we just and a series of patched merge in integrated | 19:29 |
| irtermite | jroll: at first thought you all were saying there was no v6 route through the ISP, but now it looks like you all are saying it's just a matter of poor networking on their end? | 19:29 |
| pabelanger | still ongoing | 19:29 |
| *** ddieterly[away] is now known as ddieterly | 19:31 | |
| jroll | irtermite: that's what clark said, I'm saying it didn't seem limited to v6, comcast, or PNW | 19:31 |
| openstackgerrit | Merged openstack-infra/project-config: Add After=network.target for urandom.service https://review.openstack.org/358039 | 19:31 |
| *** cody-somerville has quit IRC | 19:31 | |
| irtermite | ah, right on jroll | 19:31 |
| pabelanger | yay | 19:31 |
| cloudnull | woot! so was that the root cause? | 19:32 |
| *** roxanaghe has quit IRC | 19:32 | |
| irtermite | jroll: basically just making sure you don't think anything is broken on our end, and I have to go get networking involved to fix the environment | 19:32 |
| *** Goneri has quit IRC | 19:32 | |
| cloudnull | the unit file just had to be added after the network target? | 19:32 |
| clarkb | cloudnull: that seemed to be the case, I just got a local build to work with that patch. I don't unerstand why though | 19:33 |
| irtermite | cloudnull: you aren't here today are you? | 19:33 |
| cloudnull | because SystemD | 19:33 |
| cloudnull | :) | 19:33 |
| pabelanger | yes | 19:33 |
| * cloudnull is a helper i know... | 19:34 | |
| irtermite | #becauseSystemD | 19:34 |
| jroll | irtermite: I work in public cloud, I'd already be bugging people if it was a rackspace problem and was still ongoing :) | 19:34 |
| irtermite | jroll: I know... but I own the account. ;) | 19:34 |
| cloudnull | irtermite: I'm not at the castle | 19:34 |
| irtermite | so, you'd be bugging me | 19:34 |
| jroll | irtermite: the openstack-infra account? | 19:34 |
| irtermite | cloudnull: right on... was going to show you some happy numbers | 19:34 |
| jroll | neat :) | 19:34 |
| jroll | TIL | 19:34 |
| irtermite | jroll: no... OSIC... so, by virtue, yes infra | 19:35 |
| irtermite | all of osic | 19:35 |
| clarkb | I shoudl clarify I don't think that rackspace is at fault for the comcast ipv6 issues I described | 19:35 |
| clarkb | once I managed to complain and they foudn an asymetric route that was probably causing issues | 19:35 |
| irtermite | clarkb: no worries, we got that | 19:35 |
| clarkb | but I am not longer a comcast customer so I don't notice anymore >_> (but I also dont' get native ipv6 with new isp) | 19:36 |
| *** roxanaghe has joined #openstack-infra | 19:36 | |
| *** asettle has quit IRC | 19:38 | |
| pabelanger | Ya, I need to find an ipv6 tunnel broker myself | 19:40 |
| *** sarob has quit IRC | 19:41 | |
| clarkb | I briefly considered using dreamhosts unlimited bandwidth to set up a tunnel via them but apparently that can end at any time | 19:41 |
| cloudnull | pabelanger: https://tunnelbroker.net/ | 19:41 |
| cloudnull | they work well | 19:41 |
| *** sdake has quit IRC | 19:41 | |
| clarkb | ya there is the HE option | 19:41 |
| *** rbuzatu has quit IRC | 19:41 | |
| pabelanger | I thought you needed to be a member? | 19:41 |
| *** valderrv_ has joined #openstack-infra | 19:41 | |
| clarkb | pabelanger: you have to sign up and get certified | 19:41 |
| pabelanger | maybe not | 19:41 |
| cloudnull | just sign up | 19:41 |
| *** sarob has joined #openstack-infra | 19:41 | |
| clarkb | if you don't do the exam supposedly some ports are blocked (like irc) | 19:42 |
| *** piet has joined #openstack-infra | 19:42 | |
| krotscheck | Anyone here have opinions on whether javascript projects should be gated on devstack? :D :D https://review.openstack.org/#/c/348056/ | 19:42 |
| cloudnull | if you do the exam you get a tshirt | 19:42 |
| cloudnull | its worth it | 19:42 |
| cloudnull | :) | 19:42 |
| *** hpe-hj has quit IRC | 19:42 | |
| *** psilvad has quit IRC | 19:42 | |
| pabelanger | cloudnull: clarkb: neat, registered. I'll configure it this weekend | 19:43 |
| fungi | pabelanger: i especially like that they'll route you a /48, and delegate reverse dns anywhere you want | 19:44 |
| cloudnull | you too can beccome an IPv6 sage! ;) | 19:44 |
| *** rbuzatu has joined #openstack-infra | 19:44 | |
| *** jcoufal has quit IRC | 19:44 | |
| fungi | though updating my endpoint configuration under openbsd is a little fiddly if my isp updates my dns lease to a new ip address | 19:46 |
| fungi | i have it basically scripted | 19:46 |
| clarkb | fungi: the little updater daemon doesn't run on openbsd? | 19:46 |
| fungi | clarkb: i forget now | 19:46 |
| *** markusry has joined #openstack-infra | 19:46 | |
| clarkb | openwrt appears to support it out of the box | 19:47 |
| *** esberglu has quit IRC | 19:47 | |
| fungi | :q | 19:48 |
| *** yamamoto has joined #openstack-infra | 19:48 | |
| fungi | hah | 19:48 |
| cloudnull | ++ openwrt is great | 19:48 |
| fungi | that's not my vi window! | 19:48 |
| *** markusry has quit IRC | 19:48 | |
| *** valderrv_ has quit IRC | 19:49 | |
| clarkb | pabelanger: looks like the fix merged we need to manually update the git repo and start a build (manual since puppet is disabled?) | 19:51 |
| pabelanger | clarkb: yes, I can do that now | 19:51 |
| clarkb | ok | 19:52 |
| clarkb | hopefully with that out of the way we can sort out the ntp situation | 19:52 |
| *** pilgrimstack has joined #openstack-infra | 19:52 | |
| clarkb | do we think maybe we should just revert back to the ntpdate in d-g for now? | 19:52 |
| clarkb | while we work through to getting more platform appropriate ntp updates workign? | 19:52 |
| clarkb | fungi: ^ | 19:52 |
| *** yamamoto has quit IRC | 19:52 | |
| fungi | clarkb: i thought the consensus was that we could just remove it entirely on xenial? | 19:53 |
| fungi | and systemd would make magic with its elves | 19:53 |
| *** sarob has quit IRC | 19:53 | |
| clarkb | fungi: systemd will make magic of some sort yes but the actual magic is undocumented and who knows if it will step | 19:54 |
| *** sarob has joined #openstack-infra | 19:54 | |
| pabelanger | clarkb: applied the patch to /etc/nodepool/elements, new build started | 19:54 |
| clarkb | trusty will not though | 19:54 |
| clarkb | fungi: I am specifically thinking about hwo can we make the d-g job runs less bad | 19:54 |
| clarkb | since right now they frequently stall out | 19:54 |
| fungi | oh, are we back to hitting time skips after the timer for the job runtime has been set? | 19:55 |
| clarkb | no just job stalls | 19:55 |
| clarkb | wiating on ntp-wait | 19:55 |
| *** rfolco has quit IRC | 19:57 | |
| clarkb | thinking it may be a good idea to revert the d-g change for now then remove ntp munging from it entirely once we get per platform stuff working | 19:58 |
| *** pilgrimstack has quit IRC | 19:58 | |
| *** valderrv_ has joined #openstack-infra | 19:59 | |
| *** vinaypotluri has quit IRC | 20:01 | |
| *** xyang1 has quit IRC | 20:02 | |
| *** esberglu has joined #openstack-infra | 20:02 | |
| *** xyang1 has joined #openstack-infra | 20:02 | |
| *** esberglu has quit IRC | 20:02 | |
| *** burgerk has quit IRC | 20:03 | |
| *** xyang1 has quit IRC | 20:04 | |
| fungi | that seems fine | 20:05 |
| *** sputnik13 has quit IRC | 20:05 | |
| fungi | was the switch to ntp-wait precipitated by anything other than forward thinking? | 20:05 |
| *** sputnik13 has joined #openstack-infra | 20:05 | |
| *** sputnik13 has quit IRC | 20:06 | |
| *** jcoufal has joined #openstack-infra | 20:06 | |
| fungi | like, was there a bug it was actually addressing (like removal of ntpdate on one or more supported platforms) or merely someone spooked by the documented deprecation | 20:06 |
| *** xyang1 has joined #openstack-infra | 20:06 | |
| *** valderrv_ has quit IRC | 20:06 | |
| clarkb | oh right no ntpdate on xenial | 20:07 |
| clarkb | gah | 20:07 |
| clarkb | has to be separately installed because of the whole deprecation thing aiui | 20:07 |
| clarkb | this should be simpler :/ | 20:08 |
| *** esberglu has joined #openstack-infra | 20:08 | |
| jeblair | clarkb: ntpdate does exist on xenial though, right? | 20:09 |
| clarkb | jeblair: as a separate package yes | 20:09 |
| clarkb | whereas in older ubuntu's its part of ntp | 20:09 |
| jeblair | i'm not opposed to installing that in our base images | 20:09 |
| *** mtanino has quit IRC | 20:10 | |
| clarkb | I can add it to the infra package needs element | 20:10 |
| clarkb | will work on that now (I have to sort out what the package map is for all our distros and releases) | 20:10 |
| *** raunak has joined #openstack-infra | 20:13 | |
| *** edmondsw has quit IRC | 20:15 | |
| clarkb | ianw: btw I see Failed to synchronize cache for repo 'iwienand-kernel-f24-rhbz1361414', disabling. on our f23 builds. We probably need to scope that kernel thing to f24 | 20:16 |
| *** salv-orlando has quit IRC | 20:16 | |
| *** baoli has quit IRC | 20:17 | |
| *** roxanaghe has quit IRC | 20:17 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Ensure ntpdate is on our test images https://review.openstack.org/358104 | 20:18 |
| openstackgerrit | Clark Boylan proposed openstack-infra/devstack-gate: Revert "Use ntp-wait instead of ntpdate to set time" https://review.openstack.org/358105 | 20:18 |
| *** _sarob has joined #openstack-infra | 20:18 | |
| clarkb | ok ^ those changes along with new images should make things happier until we can properly redo the time sync on our images | 20:18 |
| *** chem` has joined #openstack-infra | 20:21 | |
| *** sarob has quit IRC | 20:21 | |
| *** chem has quit IRC | 20:22 | |
| *** rhallisey has quit IRC | 20:23 | |
| *** _sarob has quit IRC | 20:23 | |
| *** mdrabe has quit IRC | 20:24 | |
| *** esikachev has joined #openstack-infra | 20:26 | |
| *** pfallenop has quit IRC | 20:26 | |
| *** markusry has joined #openstack-infra | 20:27 | |
| *** jheroux has quit IRC | 20:29 | |
| *** pfallenop has joined #openstack-infra | 20:32 | |
| *** yamamoto has joined #openstack-infra | 20:32 | |
| *** Jeffrey4l_ has joined #openstack-infra | 20:34 | |
| *** Jeffrey4l has quit IRC | 20:35 | |
| *** dprince has quit IRC | 20:35 | |
| *** yamamoto has quit IRC | 20:37 | |
| *** roxanaghe has joined #openstack-infra | 20:38 | |
| *** gouthamr has quit IRC | 20:39 | |
| *** pfallenop has quit IRC | 20:43 | |
| jeblair | clarkb: lgtm | 20:45 |
| *** awayne has joined #openstack-infra | 20:47 | |
| *** spzala has quit IRC | 20:47 | |
| *** spzala has joined #openstack-infra | 20:48 | |
| *** zxiiro is now known as zxiiro-away | 20:48 | |
| *** amitgandhinz has quit IRC | 20:49 | |
| *** ccamacho|away has left #openstack-infra | 20:50 | |
| *** spzala has quit IRC | 20:51 | |
| *** spzala has joined #openstack-infra | 20:51 | |
| *** amitgandhinz has joined #openstack-infra | 20:51 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/project-config: Make gate-tempest-dsvm-multinode-live-migration gating for nova https://review.openstack.org/356043 | 20:51 |
| fungi | clarkb: approved the nodepool element addition, but we presumably should wait on the d-g revert until we have new images built and uploaded with that, right? | 20:55 |
| *** pfallenop has joined #openstack-infra | 20:55 | |
| *** gouthamr has joined #openstack-infra | 20:55 | |
| *** gouthamr_ has joined #openstack-infra | 21:01 | |
| *** pfallenop has quit IRC | 21:01 | |
| *** jkilpatr has quit IRC | 21:01 | |
| *** esberglu has quit IRC | 21:01 | |
| *** _coolsvap_ has quit IRC | 21:01 | |
| *** rbuzatu has quit IRC | 21:04 | |
| *** esikachev has quit IRC | 21:04 | |
| *** gouthamr has quit IRC | 21:04 | |
| *** raildo has quit IRC | 21:04 | |
| *** thiagop has quit IRC | 21:05 | |
| *** clenimar has quit IRC | 21:05 | |
| clarkb | fungi: yup | 21:06 |
| fungi | crinkle: puppet sanity check if you have a sec... http://git.openstack.org/cgit/openstack-infra/puppet-mediawiki/tree/templates/apache/mediawiki.erb#n46 asserts that undef values don't show up as nil in erb templates until puppet 4.x, and that for puppet 3.x you need to match on :undef instead... but we're still on puppet 3.8 and have been putting nil in templates for a while so i'm confused | 21:06 |
| *** sdague has quit IRC | 21:06 | |
| clarkb | fungi: at least for xenial I think on the other releases ntpdate is transitively pulled in by our ntp stuff | 21:06 |
| *** yamamoto has joined #openstack-infra | 21:07 | |
| fungi | clarkb: right, it would break devstack-gate jobs that try to run on xenial, so we at least need updated xenial images | 21:07 |
| clarkb | yup | 21:07 |
| *** jcoufal has quit IRC | 21:08 | |
| *** pfallenop has joined #openstack-infra | 21:08 | |
| *** annegentle has quit IRC | 21:10 | |
| *** yamamoto has quit IRC | 21:11 | |
| crinkle | fungi: hrm i remember putting that in but that conflicts with how i thought it worked | 21:14 |
| fungi | just wondering if it's safe to strip it down to if scope['mediawiki::whatever'] != nil | 21:15 |
| fungi | for puppet 3.8 | 21:15 |
| fungi | anyway, i need to go grab dinner before the tourist swarms descend on all our restaurants. bbiaw | 21:15 |
| *** ldnunes has quit IRC | 21:16 | |
| *** florianf has quit IRC | 21:17 | |
| crinkle | they make it so hard to find the puppet 3 documentation :( | 21:17 |
| openstackgerrit | Luigi Toscano proposed openstack-infra/project-config: Revert "Temporary allow merges for sahara-tests-release" https://review.openstack.org/357291 | 21:19 |
| *** xyang1 has quit IRC | 21:21 | |
| *** pbourke has quit IRC | 21:25 | |
| clarkb | next week assuming these test image and provider issues are mostly sorted I want to get back to xenialing things | 21:25 |
| clarkb | hopefully get that finished up over the next little bit | 21:25 |
| crinkle | fungi: it's not safe, scope[] and scope.lookupvar do use :undef in puppet 3 | 21:25 |
| crinkle | fungi: it's the @vars that use nil | 21:26 |
| pabelanger | clarkb: ya, I have some outstanding reviews for that too | 21:28 |
| mriedem | lascii: before i leave, i've left comments in https://review.openstack.org/#/c/263927/ | 21:28 |
| mriedem | oops wrong channel | 21:29 |
| *** jkilpatr has joined #openstack-infra | 21:29 | |
| *** roxanaghe has quit IRC | 21:32 | |
| *** nmagnezi has joined #openstack-infra | 21:34 | |
| *** roxanaghe has joined #openstack-infra | 21:35 | |
| *** sdake has joined #openstack-infra | 21:35 | |
| *** sdake has quit IRC | 21:37 | |
| *** sdake has joined #openstack-infra | 21:38 | |
| *** yaume has joined #openstack-infra | 21:38 | |
| *** mriedem has quit IRC | 21:38 | |
| *** thorst_ has quit IRC | 21:39 | |
| *** thorst has joined #openstack-infra | 21:41 | |
| clarkb | ugh this heat has made it hard to work | 21:41 |
| *** rbuzatu has joined #openstack-infra | 21:41 | |
| *** rbuzatu has quit IRC | 21:41 | |
| *** rbuzatu has joined #openstack-infra | 21:42 | |
| clarkb | I clearly need better AC this 17 degree delta is not enough | 21:42 |
| *** thorst has quit IRC | 21:45 | |
| *** rbuzatu has quit IRC | 21:45 | |
| bkero | clarkb: can come work over here | 21:45 |
| bkero | nice and chilly | 21:45 |
| *** rbuzatu has joined #openstack-infra | 21:46 | |
| clarkb | bkero: but that involves going outside, which I recently attempted | 21:46 |
| bkero | Need that remote start for your car so it can cool itself. | 21:46 |
| *** roxanaghe has quit IRC | 21:47 | |
| *** baoli has joined #openstack-infra | 21:47 | |
| openstackgerrit | Jay Faulkner proposed openstack-infra/project-config: Add docs jobs to ironic-lib https://review.openstack.org/356797 | 21:52 |
| *** dimtruck is now known as zz_dimtruck | 21:52 | |
| zigo | pabelanger: Still around? | 21:53 |
| *** spzala has quit IRC | 21:53 | |
| zigo | pabelanger: I attempted to rebuild deb-kazoo, and the result is that it uploaded at: http://tarballs.openstack.org/packaging-deb/deb-pkg/ | 21:53 |
| zigo | When it was supposed to upload at http://tarballs.openstack.org/packaging-deb/deb-kazoo/ | 21:53 |
| *** spzala has joined #openstack-infra | 21:54 | |
| zigo | Therefore, the pickup-job fails to wget recursively with a 404 error ... | 21:54 |
| zigo | What's wrong? | 21:54 |
| zigo | fungi: ^ | 21:54 |
| zigo | Any idea? | 21:54 |
| *** baoli has quit IRC | 21:54 | |
| clarkb | zigo: I would start by reading the job log | 21:55 |
| clarkb | zigo: do those uploads happen in the post queue? | 21:55 |
| zigo | clarkb: Yes. | 21:55 |
| pabelanger | looking | 21:55 |
| *** amitgandhinz has quit IRC | 21:56 | |
| zigo | It's at http://logs.openstack.org/dc/dc9e4dc922b65b6cb37e4bd84962a9fb035bb08e/post/ | 21:56 |
| *** thorst has joined #openstack-infra | 21:56 | |
| pabelanger | ya, I see the issue | 21:56 |
| *** gouthamr_ has quit IRC | 21:56 | |
| pabelanger | we need to fix your debpkg jobs | 21:57 |
| *** spzala has quit IRC | 21:58 | |
| clarkb | http://logs.openstack.org/dc/dc9e4dc922b65b6cb37e4bd84962a9fb035bb08e/post/pkgdeb-publish-pkg/6d595e2/_zuul_ansible/ansible_log.txt has the log for that copy in it | 21:58 |
| clarkb | you can see it goes to deb-pkg | 21:58 |
| zigo | pabelanger: Because it has "project: deb-pkg", instead of picking-up the git repo name, right? | 21:59 |
| *** thorst has quit IRC | 21:59 | |
| * zigo has no clue how to fetch the git repo name instead... :/ | 21:59 | |
| *** psachin has quit IRC | 21:59 | |
| pabelanger | zigo: yes, working on fixing it | 21:59 |
| zigo | Thanks. | 22:00 |
| zigo | I'm not sure how many beers I'll own you on the next summit... :P | 22:00 |
| *** salv-orlando has joined #openstack-infra | 22:00 | |
| *** gouthamr has joined #openstack-infra | 22:01 | |
| clarkb | ya that needs to be a template if you want it to work that way | 22:01 |
| pabelanger | yup | 22:01 |
| *** javeriak has joined #openstack-infra | 22:01 | |
| pabelanger | making it more generic while I am at it | 22:01 |
| *** lascii is now known as alaski | 22:02 | |
| *** javeriak has quit IRC | 22:06 | |
| clarkb | finally image converting the xenial iamge | 22:07 |
| *** piet has quit IRC | 22:08 | |
| pabelanger | nice | 22:08 |
| *** rlandy has quit IRC | 22:09 | |
| *** gordc has quit IRC | 22:13 | |
| *** raunak has quit IRC | 22:17 | |
| *** erlon has quit IRC | 22:17 | |
| *** _nadya_ has quit IRC | 22:22 | |
| fungi | crinkle: perfect--thanks! | 22:23 |
| *** zz_dimtruck is now known as dimtruck | 22:26 | |
| *** fguillot has joined #openstack-infra | 22:27 | |
| *** Hal1 has quit IRC | 22:28 | |
| jlvillal | Anyone else seeing the gerrit hang? | 22:28 |
| zigo | o/ | 22:28 |
| zigo | me as well... | 22:28 |
| *** mari0jv has joined #openstack-infra | 22:28 | |
| mari0jv | hi infra - i'm sure you all are aware, but a couple of others and i have been experiencing intermittent gerrit connectivity issues (review.openstack.org web) | 22:29 |
| jlvillal | Well it is a lovely sunny Friday here. Okay maybe too hot, 101F / 38C | 22:29 |
| zigo | Guys, if you've setup a monitoring for the gerrit web service, you've lost your time ... :P | 22:29 |
| zigo | A mirriad of users can play that role. | 22:30 |
| *** piet has joined #openstack-infra | 22:30 | |
| oomichi | hi, can we know bug situation in 30 days or each development cycle? | 22:30 |
| oomichi | http://status.openstack.org/bugday/ is good for me, but that seems for a few days | 22:30 |
| *** piet has quit IRC | 22:30 | |
| *** mari0jv has left #openstack-infra | 22:31 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-mediawiki: Drop Ubuntu Precise compatibility https://review.openstack.org/358084 | 22:31 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-mediawiki: Standardize HTTPS and vhost configuration https://review.openstack.org/358136 | 22:31 |
| oomichi | I'd like to see bug situation for more long-term | 22:31 |
| fungi | i guess i should check the gerrit javamelody stats | 22:31 |
| fungi | maybe we're overdue for a memory cleansing restart | 22:31 |
| clarkb | fungi: I think we did one this morning based on the graphs today | 22:32 |
| clarkb | load is relatively high on the server, gerrit looks busy | 22:32 |
| fungi | yeah, it's looking fine to me | 22:32 |
| openstackgerrit | Merged openstack-infra/system-config: Add IPv6 security group rule to cloud-launcher https://review.openstack.org/354851 | 22:32 |
| fungi | gertty and git-review haven't flinched, so i wasn't even aware | 22:32 |
| clarkb | but we have 16 vcpus | 22:32 |
| openstackgerrit | Merged openstack-infra/system-config: Add tripleo-test-cloud configuration for cloud-launcher https://review.openstack.org/354268 | 22:33 |
| fungi | oomichi: that seems fine to me. i'm not deeply familiar with the bugday codebase, but someone would probably review a patch to change the duration on it | 22:33 |
| clarkb | iowait doesn't look crazy | 22:33 |
| fungi | oomichi: the original use case for it was teams hosting a "bug day" and wanting to see how many bugs they tackled over a 24-hour period | 22:34 |
| pabelanger | clarkb: starting image upload | 22:34 |
| pabelanger | actually | 22:34 |
| pabelanger | soon(tm) starting upload | 22:34 |
| clarkb | pabelanger: its not done yet so don't use the nodepool command | 22:34 |
| clarkb | pabelanger: you can upload yourself as the qcow2 is done I htink | 22:35 |
| oomichi | fungi: yeah, current one is good for motivate people to concentrate on a bug day | 22:35 |
| clarkb | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=45&rra_id=all is interesting | 22:35 |
| oomichi | fungi: maybe it would be fine to just add some options for selecting the duration | 22:35 |
| clarkb | it looks like a dos | 22:36 |
| oomichi | fungi: thanks for your response, I will do some prototype for that | 22:36 |
| fungi | oomichi: source code is at http://git.openstack.org/cgit/openstack-infra/bugdaystats/tree/ | 22:36 |
| clarkb | but not a crippling one | 22:36 |
| fungi | clarkb: checking logs | 22:36 |
| clarkb | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=30&rra_id=all we doubled our average tcp conncetion count too | 22:37 |
| oomichi | fungi: yeah, I saw that. that is not so difficult and maybe that is better than my scratch(which I tried before) | 22:38 |
| pabelanger | clarkb: ya, I'll let DIB finish, I jumped the gun a little | 22:38 |
| *** tpsilva has quit IRC | 22:38 | |
| fungi | clarkb: some quick analysis of the sshd_log suggests it's at least not a runaway ci system looping commentspam | 22:39 |
| pabelanger | zigo: just testing the patch locally now | 22:39 |
| *** yaume has quit IRC | 22:40 | |
| *** yaume_ has joined #openstack-infra | 22:40 | |
| fungi | clarkb: zuul status shows a lengthy fuel-library change series just entered the check pipeline... could that be related? | 22:41 |
| clarkb | fungi: maybe? we could be dosing gerrit with the zuul mergers making all the zuul refs | 22:41 |
| fungi | clarkb: fuel-ci likes to add "build started" comments for every job it runs | 22:42 |
| fungi | like 9 comments on each change plus one for the fuel packaging ci, so 10 comments x 24 changes in that series | 22:43 |
| *** ddieterly is now known as ddieterly[away] | 22:43 | |
| fungi | likely all at once | 22:43 |
| *** raunak has joined #openstack-infra | 22:43 | |
| clarkb | fungi: looking at gerrit's init script it has a default fd limit of 1k which may be too small when we have that many connections. Possibly we need to tweak apache for more connections too | 22:43 |
| *** yaume_ has quit IRC | 22:44 | |
| fungi | this does not surprise me | 22:45 |
| fungi | the spike in the check sparkline graph looks like it happened right about when people started complaining | 22:47 |
| openstackgerrit | Merged openstack-infra/system-config: Add osic-cloud1 to cloud-launcher https://review.openstack.org/354852 | 22:48 |
| *** spzala has joined #openstack-infra | 22:48 | |
| *** spzala has quit IRC | 22:49 | |
| *** spzala has joined #openstack-infra | 22:49 | |
| clarkb | just checked in /proc and we have fd limit set to 8192 so tahts probably not the issue though possibly still a problem on the proxy side | 22:49 |
| *** nmagnezi has quit IRC | 22:54 | |
| *** amitgandhinz has joined #openstack-infra | 22:56 | |
| *** lbeliveau has quit IRC | 22:57 | |
| *** rbrndt has quit IRC | 22:57 | |
| *** hongbin has quit IRC | 22:58 | |
| *** salv-orlando has quit IRC | 23:00 | |
| *** spzala has quit IRC | 23:00 | |
| *** amitgandhinz has quit IRC | 23:01 | |
| *** asettle has joined #openstack-infra | 23:02 | |
| *** lbeliveau has joined #openstack-infra | 23:02 | |
| *** yamahata has quit IRC | 23:02 | |
| *** yamahata has joined #openstack-infra | 23:04 | |
| *** armax has joined #openstack-infra | 23:05 | |
| *** asettle has quit IRC | 23:06 | |
| clarkb | fungi: maxrequestworkers defaults to 400 with mpm event on apache 2.4 (if I read the docs correctly) | 23:06 |
| clarkb | we don't appear to change any of that config on review.o.o so the "slowness" could potentially be queued connections when we get a large number in all at once like that | 23:07 |
| *** dimtruck is now known as zz_dimtruck | 23:07 | |
| clarkb | we might consider bumping serverlimit/threadsperchild to increase that number | 23:07 |
| clarkb | pabelanger: waiting for vhd conversion is painful | 23:08 |
| clarkb | also I feel like these image builds have gotten much slower recently | 23:08 |
| *** amotoki has joined #openstack-infra | 23:09 | |
| pabelanger | yes, they have been slower | 23:10 |
| *** tphummel has quit IRC | 23:10 | |
| *** mattfarina has quit IRC | 23:12 | |
| *** boris-42 has quit IRC | 23:12 | |
| *** patrickeast has quit IRC | 23:12 | |
| *** raddaoui has quit IRC | 23:12 | |
| *** csatari has quit IRC | 23:13 | |
| mordred | ++ | 23:13 |
| openstackgerrit | Merged openstack-infra/project-config: Ensure ntpdate is on our test images https://review.openstack.org/358104 | 23:13 |
| pabelanger | clarkb: ubuntu-xenial uploaded | 23:15 |
| pabelanger | going to switch osic-cloud1 back online and watch | 23:16 |
| *** zz_dimtruck is now known as dimtruck | 23:16 | |
| *** mriedem has joined #openstack-infra | 23:16 | |
| *** Goneri has joined #openstack-infra | 23:17 | |
| *** dkehn_ has quit IRC | 23:17 | |
| *** dkehn has quit IRC | 23:17 | |
| *** zhurong has joined #openstack-infra | 23:18 | |
| *** raddaoui has joined #openstack-infra | 23:18 | |
| *** csatari has joined #openstack-infra | 23:19 | |
| *** patrickeast has joined #openstack-infra | 23:19 | |
| *** boris-42 has joined #openstack-infra | 23:19 | |
| *** mattfarina has joined #openstack-infra | 23:20 | |
| *** ddieterly has joined #openstack-infra | 23:22 | |
| pabelanger | nc 2001:4800:1ae1:18:f816:3eff:fe64:594e 19885 | 23:23 |
| pabelanger | is our first ubuntu-xenail | 23:23 |
| pabelanger | looks good | 23:23 |
| pabelanger | and more online now | 23:23 |
| *** amotoki has quit IRC | 23:23 | |
| pabelanger | 12 now, no failures on ready-script so far | 23:23 |
| pabelanger | I think we got it | 23:24 |
| *** dkehn has joined #openstack-infra | 23:25 | |
| pabelanger | cloudnull: ^ | 23:25 |
| pabelanger | 31 now :D | 23:25 |
| pabelanger | no, failures to launch | 23:25 |
| openstackgerrit | sandro.gattuso proposed openstack-infra/jenkins-job-builder: Wrapper pre-scm-buildstep: added support for failOnError parameter https://review.openstack.org/349219 | 23:26 |
| mordred | pabelanger: woot! | 23:27 |
| mordred | pabelanger: that's awesome | 23:27 |
| pabelanger | mordred: Ya, we have like 0 launch failures in osic-cloud1 in the last 2.5 hours | 23:27 |
| mordred | pabelanger: for multi-node, I thnk dhellmann is releasing things on monday, so when he does, we can land the new shade change, release and restart nodepool | 23:27 |
| mordred | pabelanger: that's epically awesome | 23:27 |
| pabelanger | I'm going to put ubuntu-xenial in ovh and see if that fixes the failures too | 23:28 |
| pabelanger | mordred: ++ | 23:28 |
| *** dkehn_ has joined #openstack-infra | 23:30 | |
| *** dimtruck is now known as zz_dimtruck | 23:30 | |
| *** shashank_hegde has quit IRC | 23:31 | |
| *** chem` has quit IRC | 23:31 | |
| *** chem` has joined #openstack-infra | 23:31 | |
| *** ddieterly has quit IRC | 23:32 | |
| *** markusry has quit IRC | 23:32 | |
| clarkb | nice | 23:32 |
| *** shashank_hegde has joined #openstack-infra | 23:32 | |
| *** markusry has joined #openstack-infra | 23:32 | |
| pabelanger | going to remove nodepool.o.o from emergency file | 23:32 |
| *** zhurong has quit IRC | 23:34 | |
| fungi | excellent! | 23:34 |
| *** jcoufal has joined #openstack-infra | 23:34 | |
| *** jogo has quit IRC | 23:34 | |
| fungi | clarkb: i agree given the size of review.o.o we could probably benefit from tuning maxrequestworkers | 23:34 |
| *** tosky has quit IRC | 23:34 | |
| *** jogo has joined #openstack-infra | 23:36 | |
| *** Swami has quit IRC | 23:36 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-mediawiki: Drop Ubuntu Precise compatibility https://review.openstack.org/358084 | 23:37 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-mediawiki: Standardize HTTPS and vhost configuration https://review.openstack.org/358136 | 23:37 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Refactor deb-packaging jobs https://review.openstack.org/358141 | 23:37 |
| pabelanger | zigo: ^ is your change | 23:37 |
| pabelanger | Ooh, first launch failure in osic-cloud1 | 23:38 |
| pabelanger | checking | 23:38 |
| pabelanger | dns timeout | 23:38 |
| pabelanger | lets see if it happens again | 23:39 |
| clarkb | might have uses old inage somehow? | 23:40 |
| pabelanger | I think so | 23:40 |
| pabelanger | that is what I am checking | 23:40 |
| pabelanger | no, I think it was the new image | 23:42 |
| *** dkehn has quit IRC | 23:42 | |
| *** ddieterly has joined #openstack-infra | 23:42 | |
| *** dkehn_ has quit IRC | 23:42 | |
| pabelanger | need to add some debug info into configure_mirror.sh, that include our dib build info | 23:43 |
| *** Apoorva_ has joined #openstack-infra | 23:48 | |
| *** gyee has quit IRC | 23:48 | |
| *** dkehn has joined #openstack-infra | 23:49 | |
| *** Hal has joined #openstack-infra | 23:51 | |
| *** Apoorva has quit IRC | 23:51 | |
| pabelanger | burn though the post queue now | 23:52 |
| *** Apoorva_ has quit IRC | 23:52 | |
| *** yamamoto has joined #openstack-infra | 23:53 | |
| *** Hal has quit IRC | 23:59 | |
| *** ddieterly has quit IRC | 23:59 | |
| *** Hal has joined #openstack-infra | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!