| *** mdenny has joined #openstack-infra | 00:03 | |
| *** vipul-away is now known as vipul | 00:04 | |
| *** yidclare has quit IRC | 00:04 | |
| *** blamar_ has joined #openstack-infra | 00:06 | |
| openstackgerrit | Julien Danjou proposed a change to openstack-dev/pbr: package: read a specific Python version requirement file https://review.openstack.org/63236 | 00:07 |
|---|---|---|
| jeblair | clarkb: okay, that should be the bulk of the gate queue restored | 00:07 |
| clarkb | cool there were a few config changes I had approved. I will rekick them | 00:07 |
| clarkb | oh nevermind there are a bunch of events still being processed I will wait to see if you got them :) | 00:08 |
| *** dcramer_ has quit IRC | 00:09 | |
| clarkb | there they are, patience I should learn to use it | 00:09 |
| *** blamar has quit IRC | 00:09 | |
| *** blamar_ is now known as blamar | 00:09 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/devstack-gate: WIP: test timout #2 https://review.openstack.org/62784 | 00:12 |
| openstackgerrit | A change was merged to openstack-infra/config: Use even bigger hpcloud region b nodes. https://review.openstack.org/62122 | 00:12 |
| *** gothicmindfood has quit IRC | 00:12 | |
| *** dcramer_ has joined #openstack-infra | 00:22 | |
| *** nati_ueno has joined #openstack-infra | 00:23 | |
| stevebaker | clarkb: its running again. quick, jump on it! https://jenkins02.openstack.org/job/gate-tempest-dsvm-neutron-heat-slow/6/ | 00:23 |
| stevebaker | clarkb: running tools/pretty_tox_serial.sh locally worked fine | 00:24 |
| clarkb | stevebaker: node has been held | 00:26 |
| *** nati_uen_ has quit IRC | 00:27 | |
| stevebaker | thx | 00:27 |
| clarkb | stevebaker: if you point me at an ssh rsa public key I can give you access to the box | 00:27 |
| openstackgerrit | A change was merged to openstack-infra/config: Turn on devstack exercises for neutron (non-voting) https://review.openstack.org/62930 | 00:29 |
| openstackgerrit | A change was merged to openstack-infra/config: Check for whitespace around changeid searches https://review.openstack.org/62523 | 00:29 |
| openstackgerrit | A change was merged to openstack-infra/config: Core reviewers should control WIP in Gerrit https://review.openstack.org/62428 | 00:29 |
| stevebaker | clarkb: https://launchpad.net/~steve-stevebaker/+sshkeys | 00:29 |
| *** rnirmal has joined #openstack-infra | 00:30 | |
| *** jcooley_ has quit IRC | 00:31 | |
| pleia2 | clarkb: what WM do you use instead of xfwm? | 00:31 |
| *** jcooley_ has joined #openstack-infra | 00:31 | |
| clarkb | pleia2: xmonad | 00:32 |
| pleia2 | thanks | 00:32 |
| clarkb | pleia2: it is a total hack to swap out xfwm for it though :) | 00:33 |
| pleia2 | clarkb: haha, yeah, one of the xubuntu devs took up the challenge to make it easier | 00:34 |
| *** jergerber has quit IRC | 00:34 | |
| pleia2 | (I brought it up as a joke) | 00:35 |
| *** krotscheck has joined #openstack-infra | 00:35 | |
| clarkb | nice | 00:35 |
| *** jcooley_ has quit IRC | 00:36 | |
| *** nati_ueno has quit IRC | 00:36 | |
| *** nati_ueno has joined #openstack-infra | 00:37 | |
| stevebaker | clarkb: actually, I can reproduce it locally with tox | 00:37 |
| clarkb | stevebaker: oh cool, can I unhold that slave then? | 00:38 |
| stevebaker | clarkb: release the slave! | 00:39 |
| *** odyi has quit IRC | 00:40 | |
| anteaya | stevebaker: how long have you been waiting to say that? | 00:41 |
| anteaya | that looks like a line that has been in storage for a while | 00:41 |
| stevebaker | anteaya: I should have said "free the slave" | 00:43 |
| anteaya | save that one up for next time | 00:45 |
| anteaya | there is always a next time | 00:45 |
| fungi | of course, i step away to eat, and zuul goes toes-up | 00:45 |
| fungi | sorry i missed the fun | 00:45 |
| anteaya | it was waiting for you to leave | 00:45 |
| *** odyi has joined #openstack-infra | 00:46 | |
| fungi | pleia2: i recommend the ratpoison window manager. i tried xmonad for a few weeks, but in the end decided i was just trying to reimplement all of rp inside of it and switched back | 00:46 |
| fungi | stevebaker: clarkb: technically, once a slave is no longer held, it's freed | 00:47 |
| *** Ryan_Lane has quit IRC | 00:47 | |
| openstackgerrit | A change was merged to openstack-infra/config: Switch IAD bare-precise to performance https://review.openstack.org/63242 | 00:48 |
| fungi | waiting for ^ that image to build now | 00:49 |
| *** gyee_ has joined #openstack-infra | 00:50 | |
| pleia2 | clarkb: < Unit193> pleia2: Yeah, I just switched to openbox, was pretty easy. Just edited /etc/xdg/xfce4/xfconf/xfce-perchannel-xml/xfce4-session.xml and /etc/xdg/xdg-xubuntu/xfce4/xfconf/xfce-perchannel-xml/xfce4-session.xml, but don't think you need to edit both. | 00:50 |
| clarkb | pleia2: neat, that seems better than my hack | 00:50 |
| pleia2 | then he tried xmonad and cursed a lot (because, xmonad) but it worked :) | 00:50 |
| clarkb | ha | 00:50 |
| clarkb | xmonad is lovely | 00:50 |
| fungi | i still found myself using a pointer too often in xmonad | 00:52 |
| *** dkranz has joined #openstack-infra | 00:52 | |
| fungi | i like that rp's keybindings and paradigm mostly parallel screen/tmux | 00:53 |
| clarkb | nodepool is caught up (I went ahead and deleted that slave the test had completed) | 00:54 |
| *** jcooley_ has joined #openstack-infra | 00:55 | |
| *** jcooley_ has quit IRC | 00:55 | |
| fungi | i haven't gone hunting, but i have this feeling nodepool thinks there are ready slaves on the masters which don't actually exist (based on the minimum width of the green stripe on the test nodes graph since the jenkins restarts last week) | 00:56 |
| fungi | i'll look at ages in the nodepool list | 00:56 |
| *** elasticio has joined #openstack-infra | 00:57 | |
| openstackgerrit | Ken'ichi Ohmichi proposed a change to openstack-infra/devstack-gate: Copy libvirt log file after tempest run https://review.openstack.org/61892 | 00:58 |
| fungi | roughly 20 nodes in the list with ages from 160-730 minutes so i guess not | 00:59 |
| fungi | as those were the oldest, so none over 12 hours | 00:59 |
| fungi | oh, wait, that's 160-730 HOURS | 01:00 |
| stevebaker | clarkb: so I think tox.ini setenv replaces the env rather than adds to it. | 01:00 |
| stevebaker | setenv = OS_TEST_TIMEOUT=1200 | 01:00 |
| stevebaker | that ^ was causing the problem. restating the entire env with OS_TEST_TIMEOUT=1200 at the end fixes it | 01:00 |
| clarkb | stevebaker: ah ok | 01:01 |
| stevebaker | I swear that used to work | 01:01 |
| *** nosnos has joined #openstack-infra | 01:04 | |
| fungi | okay, nodepool delete'ing 21 nodes which are >6 days old | 01:05 |
| fungi | looks like those date from the jenkins restarts or earlier | 01:06 |
| fungi | all other nodes were <1 hour old | 01:06 |
| fungi | hopefully that will increase our testing bandwidth at least a little | 01:07 |
| *** krotscheck has quit IRC | 01:08 | |
| fungi | once that finishes, i'll see whether there are any on the alien list which look like orphaned nodepool nodes | 01:08 |
| fungi | should be easy to weed out the ones from jenkins-dev since the incrementers on their names will be orders of magnitude lower | 01:09 |
| *** dstanek has joined #openstack-infra | 01:09 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Make storyboard/python26 jobs non-voting https://review.openstack.org/62443 | 01:09 |
| clarkb | that fixes a conflict going to just push it through | 01:09 |
| *** HenryG has quit IRC | 01:10 | |
| fungi | lgtm | 01:10 |
| *** CaptTofu has joined #openstack-infra | 01:12 | |
| openstackgerrit | A change was merged to openstack-infra/config: Make storyboard/python26 jobs non-voting https://review.openstack.org/62443 | 01:13 |
| *** zhiwei has joined #openstack-infra | 01:15 | |
| clarkb | mordred: https://review.openstack.org/#/c/41931/ makes me sad | 01:15 |
| *** branen has quit IRC | 01:15 | |
| mordred | clarkb: it makes me sad to | 01:15 |
| mordred | too | 01:16 |
| mordred | not to | 01:16 |
| mordred | it makes me sad two | 01:16 |
| clarkb | fungi: https://review.openstack.org/#/c/62967/ since you probably have the best grasp of that problem | 01:16 |
| fungi | to be or not to be? | 01:16 |
| openstackgerrit | Ken'ichi Ohmichi proposed a change to openstack-infra/elastic-recheck: Add query for bug 1233957 https://review.openstack.org/63259 | 01:17 |
| uvirtbot | Launchpad bug 1233957 in openstack-ci "testr_results.html.gz is not found when scp" [Undecided,New] https://launchpad.net/bugs/1233957 | 01:17 |
| *** sdake_ has quit IRC | 01:18 | |
| *** sdake__ has joined #openstack-infra | 01:18 | |
| fungi | clarkb: jeblair: looking at http://logs.openstack.org/84/62784/7/check/check-dg-tempest-dsvm-full/28d1a4d/console.html#_2013-12-20_00_39_51_969 the devstack/tempest processes are definitely disassociated from the pgroup the timeout and d-g script are in | 01:20 |
| fungi | i strongly suspect process monkeybusiness in devstack... | 01:20 |
| fungi | a new patchset with a ps afuxww would be helpful around where ps list #7 or #8 are called | 01:21 |
| fungi | clarkb: do we have any good guinea pigs you're eyeballing for trying out 62967? a change with new groups in the acl, preferably more than one group | 01:23 |
| openstackgerrit | A change was merged to openstack-infra/jeepyb: Wait for DB to reflect group UUID updates https://review.openstack.org/62967 | 01:24 |
| *** dims has quit IRC | 01:25 | |
| clarkb | fungi: I didn't have a specific one in mind but now that I think about it https://review.openstack.org/#/c/62233/4 would be good since it adds two groups | 01:25 |
| clarkb | cliff-core and cliff-ptl | 01:25 |
| mordred | clarkb: neat! nice patch | 01:25 |
| fungi | reviewing and seeing if we can approve once that's confirmed in place on review.o.o | 01:25 |
| clarkb | mordred: its about as hackish as it can get | 01:26 |
| clarkb | fungi: ok I will review the cliff change too then | 01:26 |
| clarkb | jeblair: any chance we can have you remove the -2? | 01:26 |
| *** jcooley_ has joined #openstack-infra | 01:27 | |
| fungi | if he's not around, i don't think he'll mind us deleting his -2 given we're testing to see whether we've actually resolved the reason he added it | 01:27 |
| clarkb | fungi: I agree | 01:28 |
| fungi | but it'll be a few minutes until we can test anyway | 01:29 |
| clarkb | yup I am going to run home now while we are waiting | 01:29 |
| fungi | dhellmann: if you're around, do you want to delete the "pyparsing-update" branch from the current cliff repo before we import it? | 01:31 |
| *** prad has joined #openstack-infra | 01:35 | |
| *** dims has joined #openstack-infra | 01:38 | |
| *** yaguang has joined #openstack-infra | 01:40 | |
| zhiwei | clarkb: hi | 01:41 |
| clarkb | hello | 01:41 |
| zhiwei | can I create a new stackforge project, and ask infra team to remove the old one? | 01:41 |
| zhiwei | so that this is another way for renaming project. | 01:41 |
| clarkb | no we dont rename projects that way | 01:42 |
| clarkb | I need to afk for a bit but fungi can fill you in | 01:42 |
| *** elasticio has quit IRC | 01:42 | |
| zhiwei | so, I should send an email to infra list, and ask for renaming project, then infra team will go over it? | 01:43 |
| *** melwitt has quit IRC | 01:43 | |
| fungi | we can't really delete projects from gerrit (it isn't designed for it), but it's also not great to just delete free software projects anyway | 01:44 |
| zhiwei | ok, sorry for asking that. | 01:45 |
| clarkb | but we can rename | 01:45 |
| clarkb | it just takes some effort | 01:45 |
| zhiwei | ok, thanks. I will send an email to infra list about this. Thank you very much for your quick response. | 01:46 |
| *** fifieldt has joined #openstack-infra | 01:46 | |
| fungi | zhiwei: yeah, usually project rename requests are brought up on the agenda for the weekly infra team meeting in #openstack-meeting tuesdays at 1900 utc since it requires scheduling a gerrit outage, but we're cancelling the next two meetings for holidays | 01:47 |
| zhiwei | fungi: thanks, got it. | 01:53 |
| *** matsuhashi has joined #openstack-infra | 01:58 | |
| fungi | okay, obvious devstack nodepool alien strays have been successfully nova deleted as well | 02:02 |
| *** gothicmindfood has joined #openstack-infra | 02:02 | |
| *** HenryG has joined #openstack-infra | 02:03 | |
| *** gothicmindfood has quit IRC | 02:04 | |
| clarkb | ok back | 02:07 |
| *** yongli_away is now known as yongli | 02:11 | |
| fungi | clarkb: any reason puppet isn't running on nodepool.o.o? | 02:11 |
| clarkb | fungi: not that I know of | 02:12 |
| fungi | it's no wonder that new image still aren't building | 02:12 |
| fungi | er, still isn't | 02:12 |
| clarkb | does puppet agent --test --noop show any major weird diffage? | 02:12 |
| *** jcooley_ has quit IRC | 02:12 | |
| fungi | gonna try now | 02:12 |
| fungi | nothing exciting, just our two pending nodepool config edits | 02:14 |
| fungi | starting the agent back up now | 02:14 |
| clarkb | k | 02:15 |
| openstackgerrit | A change was merged to openstack/requirements: Allow Falcon versions <0.2.0 https://review.openstack.org/62751 | 02:18 |
| *** reed has quit IRC | 02:20 | |
| *** jcooley_ has joined #openstack-infra | 02:21 | |
| *** AaronGR is now known as AaronGR_afk | 02:24 | |
| *** jergerber has joined #openstack-infra | 02:31 | |
| *** sarob_ has joined #openstack-infra | 02:31 | |
| *** jerryz has quit IRC | 02:31 | |
| *** jcooley_ has quit IRC | 02:32 | |
| *** jcooley_ has joined #openstack-infra | 02:32 | |
| *** sarob has quit IRC | 02:33 | |
| *** sarob_ has quit IRC | 02:35 | |
| jeblair | clarkb, fungi: -2 removed | 02:36 |
| fungi | jeblair: thanks! | 02:36 |
| fungi | i'll make sure review.o.o has clarkb's latest fix | 02:37 |
| *** jcooley_ has quit IRC | 02:37 | |
| fungi | yep. pip freeze reports that sha | 02:38 |
| *** praneshp has quit IRC | 02:39 | |
| *** esker has joined #openstack-infra | 02:40 | |
| *** sdake__ has quit IRC | 02:40 | |
| *** CaptTofu has quit IRC | 02:41 | |
| *** CaptTofu has joined #openstack-infra | 02:42 | |
| fungi | approved the cliff review | 02:42 |
| openstackgerrit | A change was merged to openstack-infra/config: Move cliff to stackforge https://review.openstack.org/62233 | 02:44 |
| fungi | just in time for the 0245 pulse | 02:45 |
| *** jcooley_ has joined #openstack-infra | 02:47 | |
| *** hcc has joined #openstack-infra | 02:51 | |
| *** hcc is now known as hdd_ | 02:51 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add query for neutron bug 1256106 https://review.openstack.org/63276 | 02:52 |
| uvirtbot | Launchpad bug 1256106 in neutron "Deadlock in ML2 plugin on delete port bindings" [Medium,Confirmed] https://launchpad.net/bugs/1256106 | 02:52 |
| *** jhesketh__ has quit IRC | 02:52 | |
| *** jhesketh__ has joined #openstack-infra | 02:52 | |
| *** sarob has joined #openstack-infra | 02:53 | |
| *** sarob has quit IRC | 02:59 | |
| fungi | does not seem to have helped... manage-projects finished running, but the project has no acl applied once again | 03:00 |
| fungi | checking logs | 03:00 |
| fungi | no stdout captured from the manage-projects exec in the syslog | 03:03 |
| clarkb | :( | 03:05 |
| fungi | manual run for that project returns... ERROR:manage_projects:Unable to get UUID for group cliff-ptl. | 03:05 |
| fungi | running again worked with no issue | 03:06 |
| fungi | i'll clean up/re-replicate git repos and add dhellmann to those new groups | 03:07 |
| *** dcramer_ has quit IRC | 03:08 | |
| clarkb | ok | 03:08 |
| clarkb | now that review-dev is back in action, I will have to fiddle with group creation and see what it looks like | 03:08 |
| *** sarob has joined #openstack-infra | 03:09 | |
| *** sarob_ has joined #openstack-infra | 03:10 | |
| *** vogxn has joined #openstack-infra | 03:11 | |
| *** sarob has quit IRC | 03:13 | |
| *** ladquin is now known as ladquin_afk | 03:16 | |
| *** dstanek has quit IRC | 03:19 | |
| *** dcramer_ has joined #openstack-infra | 03:21 | |
| *** sarob_ has quit IRC | 03:22 | |
| *** sarob has joined #openstack-infra | 03:22 | |
| *** rongze has joined #openstack-infra | 03:23 | |
| *** sarob_ has joined #openstack-infra | 03:26 | |
| *** sarob has quit IRC | 03:27 | |
| *** sarob_ has quit IRC | 03:28 | |
| *** sarob has joined #openstack-infra | 03:29 | |
| *** ArxCruz has quit IRC | 03:29 | |
| *** dstanek has joined #openstack-infra | 03:34 | |
| *** sarob has quit IRC | 03:40 | |
| *** sarob has joined #openstack-infra | 03:41 | |
| *** sarob has quit IRC | 03:41 | |
| *** ArxCruz has joined #openstack-infra | 03:41 | |
| *** sarob has joined #openstack-infra | 03:41 | |
| *** sdake_ has joined #openstack-infra | 03:54 | |
| *** torgomatic has left #openstack-infra | 03:56 | |
| *** jergerber has quit IRC | 04:06 | |
| *** talluri has joined #openstack-infra | 04:12 | |
| *** yaguang has quit IRC | 04:12 | |
| *** chandankumar has joined #openstack-infra | 04:15 | |
| *** ArxCruz has quit IRC | 04:15 | |
| *** harlowja is now known as harlowja_away | 04:22 | |
| *** dcramer_ has quit IRC | 04:22 | |
| *** praneshp has joined #openstack-infra | 04:25 | |
| *** prad has quit IRC | 04:28 | |
| *** AaronGR_afk is now known as AaronGr_Zzz | 04:32 | |
| *** dcramer_ has joined #openstack-infra | 04:35 | |
| *** rnirmal has quit IRC | 04:40 | |
| *** ryanpetrello has quit IRC | 04:44 | |
| *** dcramer_ has quit IRC | 04:46 | |
| *** rongze has quit IRC | 04:47 | |
| *** talluri has quit IRC | 04:48 | |
| *** jcooley_ has quit IRC | 04:48 | |
| *** jcooley_ has joined #openstack-infra | 04:49 | |
| *** talluri has joined #openstack-infra | 04:49 | |
| *** talluri has quit IRC | 04:50 | |
| *** jcooley_ has quit IRC | 04:53 | |
| *** chandankumar has quit IRC | 05:00 | |
| *** chandankumar has joined #openstack-infra | 05:01 | |
| *** boris-42 has joined #openstack-infra | 05:01 | |
| *** rongze has joined #openstack-infra | 05:03 | |
| *** rongze has quit IRC | 05:03 | |
| *** zhiwei has quit IRC | 05:06 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Set up openstackid module https://review.openstack.org/63316 | 05:13 |
| *** nati_ueno has quit IRC | 05:17 | |
| *** dstanek has quit IRC | 05:18 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Set up openstackid module https://review.openstack.org/63316 | 05:29 |
| *** SergeyLukjanov has joined #openstack-infra | 05:29 | |
| fungi | okay, that's all the fun i can stand for one night | 05:29 |
| *** rongze has joined #openstack-infra | 05:34 | |
| *** nati_ueno has joined #openstack-infra | 05:35 | |
| *** gyee_ has quit IRC | 05:35 | |
| openstackgerrit | Tom Fifield proposed a change to openstack-infra/config: Add welcome_message.py to patchset-created trigger https://review.openstack.org/61898 | 05:39 |
| *** zhiwei has joined #openstack-infra | 05:42 | |
| *** rongze has quit IRC | 05:42 | |
| *** dstanek has joined #openstack-infra | 05:47 | |
| *** jcooley_ has joined #openstack-infra | 05:48 | |
| *** sarob_ has joined #openstack-infra | 05:51 | |
| *** nicedice has quit IRC | 05:52 | |
| *** sarob has quit IRC | 05:53 | |
| *** hdd_ has quit IRC | 05:56 | |
| *** talluri has joined #openstack-infra | 05:56 | |
| *** SushilKM__ has joined #openstack-infra | 06:06 | |
| *** nati_ueno has quit IRC | 06:07 | |
| *** praneshp_ has joined #openstack-infra | 06:14 | |
| *** sarob_ has quit IRC | 06:14 | |
| *** sarob_ has joined #openstack-infra | 06:15 | |
| *** vkozhukalov has joined #openstack-infra | 06:15 | |
| *** praneshp has quit IRC | 06:16 | |
| *** praneshp_ is now known as praneshp | 06:16 | |
| *** sarob_ has quit IRC | 06:19 | |
| *** rongze has joined #openstack-infra | 06:20 | |
| *** dkliban has quit IRC | 06:29 | |
| *** SergeyLukjanov has quit IRC | 06:31 | |
| *** dkliban has joined #openstack-infra | 06:31 | |
| *** AaronGr_Zzz is now known as AaronGr | 06:40 | |
| *** nosnos has quit IRC | 06:41 | |
| *** nosnos has joined #openstack-infra | 06:42 | |
| *** esker has quit IRC | 06:42 | |
| *** dkliban has quit IRC | 06:43 | |
| *** AaronGr is now known as AaronGr_Zzz | 06:50 | |
| *** vkozhukalov has quit IRC | 06:52 | |
| *** afazekas has joined #openstack-infra | 06:57 | |
| *** yolanda has joined #openstack-infra | 06:57 | |
| *** nosnos_ has joined #openstack-infra | 07:03 | |
| *** nosnos has quit IRC | 07:05 | |
| *** NikitaKonovalov has joined #openstack-infra | 07:07 | |
| *** zhiyan has joined #openstack-infra | 07:09 | |
| *** nati_ueno has joined #openstack-infra | 07:11 | |
| *** coolsvap has joined #openstack-infra | 07:15 | |
| *** pcrews has quit IRC | 07:18 | |
| *** zhiyan has quit IRC | 07:23 | |
| *** zhiyan has joined #openstack-infra | 07:24 | |
| *** sarob_ has joined #openstack-infra | 07:25 | |
| coolsvap | lifeless: hi | 07:27 |
| *** SergeyLukjanov has joined #openstack-infra | 07:28 | |
| coolsvap | can you give me some pointers for setting up own openstack CI, I checked ci.openstack.org -> setting up your own environment | 07:28 |
| coolsvap | iw as just wondering if it is possible to set it up on a development laptop | 07:29 |
| *** sarob_ has quit IRC | 07:30 | |
| lifeless | uhm, you could, using lxc | 07:31 |
| lifeless | yes | 07:31 |
| *** nati_ueno has quit IRC | 07:31 | |
| lifeless | btw, just ask questions here - I'm no expert | 07:33 |
| coolsvap | okay sure | 07:34 |
| coolsvap | guys, I want to setup the openstack ci on my laptop for to test a poc, is it possible to set it up ? | 07:35 |
| *** yolanda has quit IRC | 07:38 | |
| lifeless | coolsvap: as I said, with lxc, probably | 07:41 |
| lifeless | coolsvap: all the dynamic stuff though depends on an openstack cloud | 07:42 |
| lifeless | so yuou'll need a cloud at some point | 07:42 |
| coolsvap | yeah, can i use devstack for that? | 07:42 |
| lifeless | yes, but again you'll probably want to use the lxc driver, for memory footprint of the vms | 07:44 |
| coolsvap | okay, let me start with that first.. using lxc driver in devstack | 07:44 |
| *** Mithrandir has quit IRC | 08:00 | |
| *** dizquierdo has joined #openstack-infra | 08:04 | |
| *** Mithrandir has joined #openstack-infra | 08:05 | |
| *** flaper87|afk is now known as flaper87 | 08:12 | |
| *** praneshp has quit IRC | 08:12 | |
| *** DinaBelova has joined #openstack-infra | 08:18 | |
| *** beagles has quit IRC | 08:19 | |
| *** fbo_away is now known as fbo | 08:20 | |
| *** b3nt_pin has joined #openstack-infra | 08:23 | |
| *** NikitaKonovalov has quit IRC | 08:25 | |
| *** NikitaKonovalov has joined #openstack-infra | 08:25 | |
| *** andreaf has quit IRC | 08:28 | |
| *** vkozhukalov has joined #openstack-infra | 08:31 | |
| *** afazekas has quit IRC | 08:38 | |
| *** yfried has joined #openstack-infra | 08:43 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Fix/improve projects_yaml_alphabetized script https://review.openstack.org/63353 | 08:47 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Sort review.projects.yaml.erb https://review.openstack.org/63354 | 08:47 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Make check-projects-yaml-alphabetized job voting https://review.openstack.org/63355 | 08:47 |
| *** rossella_s has joined #openstack-infra | 08:47 | |
| *** jcooley_ has quit IRC | 08:47 | |
| *** jcooley_ has joined #openstack-infra | 08:48 | |
| *** jcooley_ has quit IRC | 08:48 | |
| *** jcooley_ has joined #openstack-infra | 08:49 | |
| *** jcooley_ has quit IRC | 08:53 | |
| *** yassine has joined #openstack-infra | 08:54 | |
| *** thomasbiege1 has left #openstack-infra | 08:56 | |
| *** yaguang has joined #openstack-infra | 09:01 | |
| *** BobBallAway is now known as BobBall | 09:04 | |
| *** pblaho has joined #openstack-infra | 09:07 | |
| *** jpich has joined #openstack-infra | 09:08 | |
| *** dizquierdo has quit IRC | 09:08 | |
| *** Abhishek_ has joined #openstack-infra | 09:09 | |
| *** zhiwei has quit IRC | 09:10 | |
| *** matsuhashi has quit IRC | 09:12 | |
| *** matsuhashi has joined #openstack-infra | 09:12 | |
| *** bknudson has quit IRC | 09:14 | |
| *** bknudson has joined #openstack-infra | 09:15 | |
| *** matsuhashi has quit IRC | 09:16 | |
| *** romcheg has joined #openstack-infra | 09:16 | |
| *** matsuhashi has joined #openstack-infra | 09:17 | |
| *** nosnos_ has quit IRC | 09:19 | |
| *** DinaBelova has quit IRC | 09:23 | |
| *** SergeyLukjanov is now known as _SergeyLukjanov | 09:28 | |
| *** _SergeyLukjanov has quit IRC | 09:29 | |
| *** hashar has joined #openstack-infra | 09:29 | |
| *** davidhadas has quit IRC | 09:34 | |
| openstackgerrit | Masayuki Igawa proposed a change to openstack-infra/elastic-recheck: Add query for bug 1263032 https://review.openstack.org/63364 | 09:35 |
| uvirtbot | Launchpad bug 1263032 in nova "Timeout while waiting on RPC response - topic: "compute.devstack-*", RPC method: "get_console_output" info: "<unknown>"" [Undecided,New] https://launchpad.net/bugs/1263032 | 09:35 |
| *** coolsvap has quit IRC | 09:35 | |
| *** coolsvap has joined #openstack-infra | 09:35 | |
| *** matsuhashi has quit IRC | 09:41 | |
| *** rossella_s has quit IRC | 09:45 | |
| *** jcooley_ has joined #openstack-infra | 09:46 | |
| *** rossella_s has joined #openstack-infra | 09:46 | |
| *** jooools has joined #openstack-infra | 09:52 | |
| *** SergeyLukjanov has joined #openstack-infra | 09:53 | |
| *** davidhadas has joined #openstack-infra | 09:57 | |
| *** derekh has joined #openstack-infra | 10:03 | |
| *** yaguang has quit IRC | 10:10 | |
| *** boris-42 has quit IRC | 10:12 | |
| *** andreaf has joined #openstack-infra | 10:14 | |
| *** jcooley_ has quit IRC | 10:16 | |
| *** jcooley_ has joined #openstack-infra | 10:16 | |
| *** yolanda_ has joined #openstack-infra | 10:27 | |
| *** NikitaKonovalov has quit IRC | 10:27 | |
| *** dizquierdo has joined #openstack-infra | 10:43 | |
| *** rongze has quit IRC | 10:44 | |
| openstackgerrit | Thierry Carrez proposed a change to openstack-infra/releasestatus: Add support for "Not" blueprint priority https://review.openstack.org/63375 | 10:45 |
| ttx | jeblair: fix for the issue you reported ^ | 10:46 |
| *** jcooley_ has quit IRC | 10:49 | |
| *** NikitaKonovalov has joined #openstack-infra | 10:49 | |
| *** rongze has joined #openstack-infra | 10:53 | |
| *** rongze has quit IRC | 11:01 | |
| *** coolsvap has quit IRC | 11:03 | |
| *** mattymo has quit IRC | 11:07 | |
| *** DinaBelova has joined #openstack-infra | 11:10 | |
| *** SergeyLukjanov is now known as _SergeyLukjanov | 11:13 | |
| *** _SergeyLukjanov has quit IRC | 11:13 | |
| *** jcooley_ has joined #openstack-infra | 11:16 | |
| *** rongze has joined #openstack-infra | 11:23 | |
| *** lcestari has joined #openstack-infra | 11:24 | |
| *** Abhishek_ has quit IRC | 11:28 | |
| *** ArxCruz has joined #openstack-infra | 11:30 | |
| *** zhiyan has quit IRC | 11:41 | |
| *** jcooley_ has quit IRC | 11:46 | |
| *** romcheg has left #openstack-infra | 11:49 | |
| *** smarcet has joined #openstack-infra | 11:56 | |
| *** SushilKM__ has left #openstack-infra | 11:58 | |
| *** smarcet has quit IRC | 12:00 | |
| *** SergeyLukjanov has joined #openstack-infra | 12:03 | |
| *** SergeyLukjanov is now known as _SergeyLukjanov | 12:03 | |
| *** gaelL has quit IRC | 12:04 | |
| *** vladan has quit IRC | 12:04 | |
| *** uvirtbot has quit IRC | 12:05 | |
| *** _SergeyLukjanov has quit IRC | 12:05 | |
| *** uvirtbot has joined #openstack-infra | 12:05 | |
| *** vladan has joined #openstack-infra | 12:06 | |
| *** yassine has quit IRC | 12:07 | |
| *** SushilKM__ has joined #openstack-infra | 12:07 | |
| *** yamahata has quit IRC | 12:09 | |
| *** SergeyLukjanov has joined #openstack-infra | 12:09 | |
| *** gaelL has joined #openstack-infra | 12:09 | |
| *** yassine has joined #openstack-infra | 12:10 | |
| *** davidhadas_ has joined #openstack-infra | 12:10 | |
| *** davidhadas has quit IRC | 12:10 | |
| *** Hefeweizen has quit IRC | 12:11 | |
| *** Hefeweizen has joined #openstack-infra | 12:11 | |
| *** SushilKM__ has quit IRC | 12:12 | |
| *** SushilKM__ has joined #openstack-infra | 12:13 | |
| *** rossella_s has quit IRC | 12:15 | |
| *** b3nt_pin has quit IRC | 12:15 | |
| *** boris-42 has joined #openstack-infra | 12:16 | |
| *** SushilKM__ has quit IRC | 12:19 | |
| *** b3nt_pin has joined #openstack-infra | 12:19 | |
| *** zhiyan has joined #openstack-infra | 12:19 | |
| *** yfried has quit IRC | 12:20 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Initial test set added https://review.openstack.org/61120 | 12:20 |
| *** smarcet has joined #openstack-infra | 12:20 | |
| *** rongze has quit IRC | 12:21 | |
| *** SushilKM__ has joined #openstack-infra | 12:23 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for bug 1263032 https://review.openstack.org/63364 | 12:23 |
| uvirtbot | Launchpad bug 1263032 in nova "Timeout while waiting on RPC response - topic: "compute.devstack-*", RPC method: "get_console_output" info: "<unknown>"" [High,Confirmed] https://launchpad.net/bugs/1263032 | 12:23 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for neutron bug 1256106 https://review.openstack.org/63276 | 12:23 |
| uvirtbot | Launchpad bug 1256106 in neutron "Deadlock in ML2 plugin on delete port bindings" [Medium,Confirmed] https://launchpad.net/bugs/1256106 | 12:23 |
| *** adalbas has quit IRC | 12:24 | |
| *** chandankumar_ has joined #openstack-infra | 12:25 | |
| *** chandankumar is now known as ciypro | 12:29 | |
| *** SushilKM__ has quit IRC | 12:30 | |
| *** SushilKM__ has joined #openstack-infra | 12:32 | |
| *** jcooley_ has joined #openstack-infra | 12:34 | |
| *** SushilKM__ has quit IRC | 12:39 | |
| *** yolanda_ has quit IRC | 12:39 | |
| *** yolanda_ has joined #openstack-infra | 12:40 | |
| *** DinaBelova has quit IRC | 12:41 | |
| *** SushilKM__ has joined #openstack-infra | 12:41 | |
| *** NikitaKonovalov has quit IRC | 12:43 | |
| *** SushilKM__ has quit IRC | 12:46 | |
| *** prad has joined #openstack-infra | 12:49 | |
| *** ruhe has joined #openstack-infra | 12:49 | |
| *** SushilKM__ has joined #openstack-infra | 12:51 | |
| *** jcooley_ has quit IRC | 12:52 | |
| *** thomasbiege has joined #openstack-infra | 12:54 | |
| *** smarcet has quit IRC | 12:58 | |
| *** NikitaKonovalov has joined #openstack-infra | 13:01 | |
| *** CaptTofu has quit IRC | 13:13 | |
| *** rossella_s has joined #openstack-infra | 13:13 | |
| *** CaptTofu_ has joined #openstack-infra | 13:13 | |
| *** rongze has joined #openstack-infra | 13:17 | |
| *** jcooley_ has joined #openstack-infra | 13:18 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Stories and Tasks search https://review.openstack.org/60515 | 13:19 |
| *** b3nt_pin is now known as beagles | 13:20 | |
| *** thomasbiege has left #openstack-infra | 13:35 | |
| *** adalbas has joined #openstack-infra | 13:37 | |
| *** prad has quit IRC | 13:38 | |
| *** boris-42 has quit IRC | 13:41 | |
| *** jcooley_ has quit IRC | 13:50 | |
| *** julim has joined #openstack-infra | 13:51 | |
| *** dizquierdo has quit IRC | 13:52 | |
| *** dprince has joined #openstack-infra | 13:53 | |
| *** ruhe is now known as ruhe_ | 13:53 | |
| *** ruhe_ is now known as ruhe | 13:53 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Set up openstackid module https://review.openstack.org/63316 | 13:54 |
| *** ryanpetrello has joined #openstack-infra | 13:57 | |
| *** bknudson has quit IRC | 13:58 | |
| *** tma996 has joined #openstack-infra | 13:58 | |
| *** NikitaKonovalov has quit IRC | 13:58 | |
| *** talluri has quit IRC | 13:59 | |
| *** talluri has joined #openstack-infra | 13:59 | |
| *** fbo is now known as fbo_away | 13:59 | |
| *** fbo_away is now known as fbo | 14:00 | |
| *** andreaf has quit IRC | 14:01 | |
| *** smarcet has joined #openstack-infra | 14:01 | |
| *** NikitaKonovalov has joined #openstack-infra | 14:02 | |
| *** dkliban has joined #openstack-infra | 14:02 | |
| *** talluri has quit IRC | 14:04 | |
| *** DinaBelova has joined #openstack-infra | 14:06 | |
| *** eharney has joined #openstack-infra | 14:07 | |
| *** mfer has joined #openstack-infra | 14:09 | |
| *** tma996_ has joined #openstack-infra | 14:09 | |
| *** tma996 has quit IRC | 14:11 | |
| *** SergeyLukjanov is now known as _SergeyLukjanov | 14:11 | |
| *** _SergeyLukjanov has quit IRC | 14:12 | |
| *** NikitaKonovalov has quit IRC | 14:14 | |
| *** yolanda_ has quit IRC | 14:15 | |
| *** derekh has quit IRC | 14:16 | |
| *** fbo has quit IRC | 14:16 | |
| *** SergeyLukjanov has joined #openstack-infra | 14:16 | |
| *** yolanda_ has joined #openstack-infra | 14:16 | |
| *** che-arne has joined #openstack-infra | 14:16 | |
| *** fbo has joined #openstack-infra | 14:17 | |
| *** dcramer_ has joined #openstack-infra | 14:18 | |
| *** prad has joined #openstack-infra | 14:19 | |
| *** bknudson has joined #openstack-infra | 14:24 | |
| *** prad has quit IRC | 14:25 | |
| *** SushilKM__ has left #openstack-infra | 14:25 | |
| *** dcramer_ has quit IRC | 14:26 | |
| *** DinaBelova has quit IRC | 14:27 | |
| *** DinaBelova has joined #openstack-infra | 14:27 | |
| HenryG | I am subscribed to some gerrit review queues, i.e. getting emails with review activity. I have a question about the message headers. Is this the right channel to ask? | 14:29 |
| *** jcooley_ has joined #openstack-infra | 14:29 | |
| fungi | HenryG: as good a channel as any i guess | 14:29 |
| SergeyLukjanov | HenryG, hey, I think that's right channel to ask about it | 14:29 |
| HenryG | thanks. Is there a way to see from the headers which comments are from automated systems? | 14:31 |
| fungi | HenryG: unfortunately not... automated systems still just use gerrit accounts like people do (the acls and group membership differ for some of them, but none of that makes it into the e-mail headers) | 14:32 |
| *** NikitaKonovalov has joined #openstack-infra | 14:32 | |
| fungi | i've simply played whack-a-mole with matching substrings in the message body to split them for my own mail filters | 14:32 |
| HenryG | fungi: whack-a-mole is exactly what I am doing :( | 14:33 |
| *** basha has joined #openstack-infra | 14:33 | |
| *** pblaho has quit IRC | 14:33 | |
| fungi | HenryG: i do this... http://paste.openstack.org/show/55680/ | 14:34 |
| fungi | and then a i have a cron job auto-expire the contents of my reviewspam maildir when they reach a week, and only have mutt notify me of messages arriving in the reviews maildir instead | 14:36 |
| HenryG | fungi: right. The neutron queue is worse. 3rd party jenkinses are coming online. | 14:36 |
| fungi | HenryG: yeah, i expect nova is feeling similar pain now | 14:36 |
| HenryG | Could we ask the automated system owners to put "Jenkins" in their name? E.g. instead of "Mellanox External Testing" please call yourself "Jenkins for Mellanox" or something? | 14:37 |
| *** chandankumar_ has quit IRC | 14:38 | |
| *** prad has joined #openstack-infra | 14:39 | |
| fungi | it's possible that we could come up with a consistent pattern to prefix/suffix the full names of those accounts so as to ease filtering | 14:39 |
| fungi | i can edit them fairly easily in the accounts table with an update query if we manage to agree on something | 14:39 |
| *** basha has quit IRC | 14:40 | |
| *** CaptTofu_ has quit IRC | 14:40 | |
| fungi | we've been wanting to rename the Jenkins user anyway because it's not really jenkins posting those comments either, so i don't think "jenkins" is a good prefix... but maybe their names could all end in "bot" or something similarly short and descriptive | 14:40 |
| HenryG | fungi: OK, sounds cool. Thanks for taking interest in my rather insignificant little problem. :) | 14:42 |
| fungi | leading to "OpenStack Zuul Bot has posted comments..." and "IBM DB2 Bot has submitted..." and so on | 14:42 |
| *** CaptTofu has joined #openstack-infra | 14:42 | |
| fungi | HenryG: it's far from insignificant. the lack of good filtering options for gerrit e-mail has led a lot of reviewers to just give up on tracking messages from it entirely | 14:43 |
| *** talluri has joined #openstack-infra | 14:43 | |
| *** thomasbiege1 has joined #openstack-infra | 14:43 | |
| fungi | i'll file a bug and get some input | 14:44 |
| HenryG | let me know the bug number and I'll watch it | 14:45 |
| ttx | fungi: could use your help to understand why I reliably hit bug 1263086 in https://review.openstack.org/#/c/62691/ testing | 14:47 |
| uvirtbot | Launchpad bug 1263086 in openstack-ci "tempest jobs fail with "ImportError: No module named config"" [Undecided,New] https://launchpad.net/bugs/1263086 | 14:47 |
| *** derekh has joined #openstack-infra | 14:47 | |
| ttx | since I hit the same issue on every test and every recheck, it's probably something in my code but i can't see it | 14:47 |
| *** afazekas_ has joined #openstack-infra | 14:49 | |
| ttx | sdague: could use your expertise too, in case that rings a bell ^ | 14:51 |
| fungi | HenryG: bug 1263121 | 14:52 |
| uvirtbot | Launchpad bug 1263121 in openstack-ci "Gerrit external testing tools should have consistent names" [Medium,Confirmed] https://launchpad.net/bugs/1263121 | 14:52 |
| fungi | ttx: having a look-see | 14:52 |
| *** dizquierdo has joined #openstack-infra | 14:52 | |
| ttx | fungi: business summary: all tempest run fail with various "ImportError: No module named config" when trying to use oslo.config -- although it looks installed alright from the logs | 14:53 |
| ttx | would blame some kind of transient issue but I hit it VERY reliably so it must be something in my change, or something my change triggers | 14:54 |
| fungi | ttx: the traceback looks truncated there | 14:55 |
| *** dcramer_ has joined #openstack-infra | 14:56 | |
| sdague | ttx: oslo.config looks like it's correct there | 14:56 |
| fungi | oh, i see the excetpion name is mentioned further up | 14:56 |
| ttx | yep, in several occurences too | 14:56 |
| *** jcooley_ has quit IRC | 14:59 | |
| *** dstanek has quit IRC | 15:00 | |
| fungi | i don't immediately see how anything in that change could be a direct cause, aside from the actual switching to oslo-rootwrap itself | 15:00 |
| *** jcooley_ has joined #openstack-infra | 15:01 | |
| *** boris-42 has joined #openstack-infra | 15:02 | |
| ttx | fungi: the only vaguely related thing is that those ImportError are when *-manage are executed under sudo | 15:02 |
| ttx | which should definitely not impact where it looks for python libs | 15:02 |
| *** yolanda_ has quit IRC | 15:02 | |
| ttx | notr care about oslo-rootwrap | 15:03 |
| fungi | i'm vaguely remembering a related thread on the -dev ml. hunting it down now | 15:03 |
| fungi | ttx: http://lists.openstack.org/pipermail/openstack-dev/2013-December/022586.html is the same exception... | 15:04 |
| dhellmann | ttx: sudo wipes some env vars, including the python library path | 15:04 |
| ttx | dhellmann: right, but the change doesn't change anything related to that. Those -manage thing were always run under sudo alright | 15:04 |
| dhellmann | ttx: fair enough | 15:05 |
| ttx | dhellmann: also it wipes the user-provided PYTHON_PATH, but then you get the system default | 15:06 |
| *** yolanda_ has joined #openstack-infra | 15:06 | |
| *** dstanek has joined #openstack-infra | 15:06 | |
| dhellmann | I thought it might be an interaction of sudo and virtualenv | 15:06 |
| dhellmann | but if the other command was run with sudo in a virtualenv too, I guess that's not related | 15:06 |
| ttx | dhellmann: if it were failing everywhere, it could be, but apparently it's only failing for my change :) | 15:07 |
| *** CaptTofu has quit IRC | 15:07 | |
| *** CaptTofu has joined #openstack-infra | 15:07 | |
| *** rnirmal has joined #openstack-infra | 15:09 | |
| *** jhesketh has quit IRC | 15:10 | |
| *** jhesketh__ has quit IRC | 15:10 | |
| *** che-arne has quit IRC | 15:12 | |
| *** Adri2000 has quit IRC | 15:13 | |
| *** che-arne has joined #openstack-infra | 15:15 | |
| sdague | ttx: when did oslo.rootwrap become a seperate thing? | 15:15 |
| sdague | I don't see it in devstack setup | 15:15 |
| ttx | sdague: I pushed a change to support it in lib/cinder | 15:15 |
| sdague | ttx: so that's wrong | 15:16 |
| ttx | sdague: explain | 15:16 |
| sdague | look at how oslo.config and olso.messaging are done in devstack / devstack-gate | 15:17 |
| *** smarcet has quit IRC | 15:17 | |
| ttx | ooh. lib/oslo | 15:18 |
| *** smarcet has joined #openstack-infra | 15:18 | |
| ttx | sdague: I don't see why that would generate that failure, but ok, looks like I should not rely on requirements.txt to get it for me ? | 15:19 |
| sdague | if an oslo library is graduated, it needs to be devstack gating on every commit | 15:20 |
| sdague | it actually needs to be that before it graduates :) | 15:20 |
| *** thomasbiege1 has quit IRC | 15:20 | |
| sdague | it should never have gotten approved in requirements until it did | 15:20 |
| ttx | sdague: ah, interesting. dhellmann and markmc did not mention that :) | 15:20 |
| * dhellmann didn't know | 15:20 | |
| dhellmann | is this stuff written down somewhere? | 15:21 |
| ttx | dhellmann: cool, I feel less stupid now | 15:21 |
| sdague | markmc should have | 15:21 |
| sdague | because we did this for oslo.messaging not that long ago | 15:21 |
| dhellmann | he might not have been involved in approving the requirements patch | 15:21 |
| sdague | he was not | 15:21 |
| dhellmann | I'd have to look at that history | 15:21 |
| dhellmann | ok | 15:21 |
| *** che-arne has quit IRC | 15:21 | |
| sdague | ok, so graduating an oslo bit means it needs to go in the integrated gate | 15:22 |
| * dhellmann reads the oslo section of the wiki | 15:22 | |
| sdague | otherwise, terribleness | 15:22 |
| sdague | because you can push a broken release to pip, and wedge the gate | 15:22 |
| dhellmann | we should add details to https://wiki.openstack.org/wiki/Oslo#Graduation | 15:22 |
| sdague | probably | 15:22 |
| dhellmann | a checklist would be awesome | 15:22 |
| sdague | there are only so many oslo extra tasks I can take on before x-mass :) | 15:23 |
| ttx | sdague: since nobody uses it yet, damage is limited | 15:23 |
| sdague | ttx: correct | 15:23 |
| sdague | but before anyone does, lets get all that setup | 15:23 |
| *** jhesketh has joined #openstack-infra | 15:23 | |
| sdague | mirror how oslo.messaging works | 15:23 |
| *** smarcet has quit IRC | 15:23 | |
| *** NikitaKonovalov has quit IRC | 15:23 | |
| dhellmann | sdague: if I could get the details out of your head directly, I could update the wiki myself :-) | 15:23 |
| *** jhesketh__ has joined #openstack-infra | 15:23 | |
| sdague | dhellmann: sure | 15:23 |
| dhellmann | sdague: where do we look for changes? devstack itself, openstack-infra/config, elsewhere? | 15:24 |
| ttx | dhellmann: was about to ask same question | 15:24 |
| sdague | devstack lib/oslo for examples there | 15:24 |
| sdague | and let me find the right config url | 15:24 |
| dhellmann | ttx: do you want to take a stab at a first draft of a checklist? | 15:24 |
| *** gothicmindfood has joined #openstack-infra | 15:25 | |
| sdague | https://github.com/openstack-infra/config/blob/master/modules/openstack_project/files/zuul/layout.yaml#L1060 | 15:25 |
| ttx | dhellmann: sure | 15:26 |
| dhellmann | ttx: thanks! | 15:26 |
| ttx | sdague: cool thx. could you explain the lib/oslo bit to me ? like why it's needed ? | 15:26 |
| sdague | ttx: if you are an integrated project, you need to be part of the integrated gate | 15:27 |
| ttx | that part I understand | 15:27 |
| sdague | so that when we have a 30 deep zuul pipeline | 15:27 |
| dhellmann | ttx: that installs the oslo bits from master instead of the latest packaged release | 15:27 |
| sdague | and it's nova, nova, cinder, keystone, oslo.rootwrap, cinder, nova ... | 15:28 |
| sdague | we are actually testing the zuul refs of all those those | 15:28 |
| *** CaptTofu has quit IRC | 15:28 | |
| *** AaronGr_Zzz is now known as AaronGr | 15:28 | |
| sdague | basically all the magic of zuul building the speculation only works if you are pulling refs out gerrit | 15:28 |
| dhellmann | sdague: what sets OSLOCFG_BRANCH and OSLOMSG_BRANCH? zuul? | 15:28 |
| sdague | yes | 15:29 |
| dhellmann | how does it know the names of the variables to use? | 15:29 |
| sdague | devstack gate actually | 15:29 |
| dhellmann | so if we add OSLOROOTWRAP_BRANCH, where do we need to add something to set it? | 15:29 |
| sdague | oh, you need to also add here - https://github.com/openstack-infra/devstack-gate/blob/master/devstack-vm-gate-wrap.sh#L396 | 15:30 |
| sdague | branching is automatic in devstack-gate | 15:30 |
| sdague | OSLOROOTWRAP_BRANCH is actually only used by devstack front end users, the branch manip in devstack-gate is it's own thing | 15:30 |
| dhellmann | ah, ok, that makes more sense | 15:31 |
| sdague | but we need the support in devstack to do the setup for the project | 15:31 |
| ttx | yep, that's set in stackrc | 15:32 |
| ttx | (OSLOCFG_BRANCH) | 15:32 |
| sdague | yep | 15:32 |
| sdague | in zuul + gate all that is actually done outside of devstack | 15:32 |
| dhellmann | sdague: I wonder if we could get that PROJECTS list from something we set up in devstack itself to cut down on the number of places we have to edit to add new projects | 15:32 |
| sdague | dhellmann: possibly, it's a low priority refactoring | 15:32 |
| dhellmann | sdague: sure | 15:33 |
| sdague | we only add a couple a cycle | 15:33 |
| *** blamar has quit IRC | 15:33 | |
| ttx | sdague: OK, I think I got it all. Will add to wiki so that I don't forget, and probably do it after vacation | 15:33 |
| sdague | cool | 15:33 |
| dhellmann | thanks, ttx | 15:33 |
| ttx | sdague: nobody will accidentally switch to solo-rootwrap in the mean time, to be sure | 15:33 |
| dhellmann | and sdague :-) | 15:33 |
| sdague | :) | 15:33 |
| ttx | it's a rather complex change | 15:33 |
| ttx | I also need to push a grenade thing to migrate the config | 15:34 |
| ttx | (a.k.a. "throw a grenade") | 15:34 |
| ttx | sdague: thx, I had no clue. | 15:34 |
| sdague | ttx: yeh no worries | 15:34 |
| sdague | honestly, markmc and I went through this all on oslo.messaging back in august, and it hadn't occured to me that it was just tribal knowledge that didn't get written down | 15:35 |
| sdague | I often judge poorly on what is common knowledge apparently | 15:36 |
| dhellmann | sdague: yeah, I suspect if he was still ptl it would have been on his mind and handled | 15:36 |
| dhellmann | I always assume instructions need to be written down, fwiw | 15:36 |
| dhellmann | if only so I can delegate :-) | 15:36 |
| dhellmann | "go follow these steps and let me know when you're ready for review" | 15:36 |
| *** blamar has joined #openstack-infra | 15:36 | |
| *** dizquierdo has quit IRC | 15:37 | |
| *** rcleere has joined #openstack-infra | 15:39 | |
| sdague | yeh, I'm always terrible at that | 15:40 |
| *** markmcclain has quit IRC | 15:41 | |
| *** jlibosva has joined #openstack-infra | 15:42 | |
| dhellmann | sdague: takes practice :-) | 15:42 |
| anteaya | fungi: if you are around | 15:43 |
| anteaya | jlibosva is still working on https://review.openstack.org/#/c/58695/ | 15:43 |
| sdague | now I'm trying to replicate the grenade explode because of the service thing | 15:43 |
| anteaya | and he is reporting to me that turning on exercises for neutron don't give him the information he is hoping to see | 15:44 |
| anteaya | can we try to accesss a running vm? | 15:44 |
| *** che-arne has joined #openstack-infra | 15:45 | |
| *** boris-42 has quit IRC | 15:45 | |
| *** jergerber has joined #openstack-infra | 15:45 | |
| sdague | anteaya: are exercises passing on neutron in devstack outside of grenade for him? | 15:45 |
| *** boris-42 has joined #openstack-infra | 15:45 | |
| *** jpich has quit IRC | 15:47 | |
| jlibosva | sdague: no | 15:48 |
| sdague | jlibosva: so start there first | 15:48 |
| jlibosva | sdague: seems like the same issue | 15:48 |
| ttx | dhellmann: Added a few pointers at https://wiki.openstack.org/wiki/Oslo#Graduation | 15:48 |
| sdague | jlibosva: yes, it will be | 15:48 |
| fungi | anteaya: jlibosva: i wrote instructions on how to make a running vm for those sorts of tests https://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/README.rst#n100 | 15:48 |
| sdague | you need the exercises passing on *both* the havana and master trees | 15:48 |
| jlibosva | fungi: thanks | 15:49 |
| sdague | then look at the grenade changes | 15:49 |
| ttx | sdague: is there a specific order in which to push the changes to succeed ? devstaack, then config, then devstack-gate ? | 15:49 |
| ttx | spaaace | 15:49 |
| sdague | devstack, devstack-gate, then config | 15:50 |
| sdague | I think | 15:50 |
| fungi | ttx: are you in space? i'm in space | 15:50 |
| ttx | no I'm in devstaack | 15:50 |
| sdague | actually, d-g => config => devstack might make it self testing on the last part | 15:50 |
| *** talluri_ has joined #openstack-infra | 15:50 | |
| *** talluri has quit IRC | 15:50 | |
| *** blamar has quit IRC | 15:53 | |
| *** dims has quit IRC | 15:54 | |
| *** mestery has quit IRC | 15:54 | |
| *** dims has joined #openstack-infra | 15:54 | |
| anteaya | jlibosva: do you have what you need? | 15:54 |
| jlibosva | anteaya: I don't know yet :) I'm reading the provided document. I'll try to simulate the test according that document | 15:55 |
| anteaya | jlibosva: okay | 15:56 |
| anteaya | fungi: thanks for writing that | 15:56 |
| anteaya | sdague: just thanks | 15:56 |
| *** blamar has joined #openstack-infra | 15:56 | |
| *** jooools has quit IRC | 15:57 | |
| sdague | fungi: how much of christmas are you off? | 15:57 |
| *** smarcet has joined #openstack-infra | 15:57 | |
| fungi | anteaya: it's basically the steps i use to run devstack-gate tests in private for embargoed security fixes (the rest of the vmt follows a similar pattern) | 15:57 |
| fungi | sdague: eh, whenever i have to be (when i'm in the car and maybe a little off and on during the week) | 15:58 |
| anteaya | k, thanks | 15:58 |
| sdague | I finally have a decent set of scripts to build me devstack nodes on rax for testing | 15:58 |
| *** dhellmann is now known as dhellmann_ | 15:58 | |
| sdague | at some point we should probably compare notes | 15:58 |
| sdague | and figure out if we could consolidate into something people could use | 15:59 |
| *** BobBall is now known as BobBallAway | 15:59 | |
| *** yolanda_ has quit IRC | 16:00 | |
| fungi | sdague: definitely. would be nice to just get it into some additional tooling in devstack-gate so we will manage to keep it in sync with the stuff it's calling | 16:00 |
| fungi | since right now i end up setting some d-g envvars which could change out from under me over time (as we discovered with one of the gate hooks for requirements integration tests after the most recent d-g refactoring) | 16:01 |
| fungi | i've distilled my workflow down to something which looks like '. ready rs myaccount iad testserver && ./set && ./go keystone master tempest.sh fixes/embargo/cve-XXXX-YYYY-master-icehouse.patch' | 16:04 |
| *** pcrews has joined #openstack-infra | 16:05 | |
| sdague | cool | 16:05 |
| annegentle_ | fungi: you might know, what bit of code do I edit to add another project to http://status.openstack.org/bugday/? | 16:05 |
| fungi | where "ready" exports the novaclient and related vars into my environment, "set" launches and configures the vm, and "go" uploads the patch i want to test and runs a stub script which sets job-specific variables (and then i have several of those depending on the job) | 16:05 |
| *** yolanda_ has joined #openstack-infra | 16:05 | |
| fungi | i could probably extend the "go" script to parse out job builder sections from the jjb yaml tree | 16:06 |
| fungi | though i think ttx has similar scripts which may already do that | 16:07 |
| fungi | annegentle_: i'll hunt it down | 16:09 |
| ttx | fungi: bugday is still generated by some script on old-wiki | 16:10 |
| fungi | annegentle_: it's something which is still | 16:10 |
| fungi | what ttx just said | 16:10 |
| ttx | annegentle_: I can add stuff to it | 16:10 |
| *** homeless has quit IRC | 16:10 | |
| *** nicedice has joined #openstack-infra | 16:10 | |
| *** homeless has joined #openstack-infra | 16:11 | |
| ttx | annegentle_: anything you'd like to see ? | 16:11 |
| ttx | last offer before next year | 16:11 |
| annegentle_ | ttx: openstack-api-site | 16:11 |
| fungi | ttx: do we have an open bug tracking that? if not i'll open one and mention it on the bug for tearing down old-wiki so we don't gotget to do that first | 16:11 |
| annegentle_ | ttx: woo last offer! winner winner | 16:11 |
| ttx | fungi: yes we have | 16:11 |
| fungi | okay, perfect | 16:11 |
| fungi | heh... "gotget" | 16:12 |
| *** praneshp has joined #openstack-infra | 16:12 | |
| fungi | my left hand must have shifted right one key | 16:12 |
| ttx | hhmm | 16:12 |
| *** tma996_ has quit IRC | 16:12 | |
| ttx | https://bugs.launchpad.net/openstack-ci/+bug/1172440 | 16:14 |
| uvirtbot | Launchpad bug 1172440 in openstack-ci "bug day stats should be in openstack-infra" [Medium,Triaged] | 16:14 |
| *** Adri2000 has joined #openstack-infra | 16:14 | |
| ttx | annegentle_: any title ? like openstack-manuals has "Manuals" ? | 16:16 |
| annegentle_ | ttx: API Site | 16:16 |
| *** smarcet has quit IRC | 16:17 | |
| annegentle_ | ttx: and thx! | 16:17 |
| *** jcooley_ has quit IRC | 16:17 | |
| *** jcooley_ has joined #openstack-infra | 16:18 | |
| *** DinaBelova has quit IRC | 16:21 | |
| sdague | dhellmann_: can you take a look at - https://review.openstack.org/#/c/63444/ ? | 16:22 |
| sdague | that's my attempted work around | 16:22 |
| *** fifieldt has quit IRC | 16:22 | |
| *** vkozhukalov has quit IRC | 16:23 | |
| *** gothicmindfood has quit IRC | 16:27 | |
| *** gothicmindfood has joined #openstack-infra | 16:29 | |
| *** smarcet has joined #openstack-infra | 16:31 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Make savanna d-g jobs voting https://review.openstack.org/63447 | 16:31 |
| dhellmann_ | sdague: I've added the Fengqian and Alessandro to the review | 16:32 |
| *** dhellmann_ is now known as dhellmann | 16:32 | |
| sdague | dhellmann: cool | 16:32 |
| dhellmann | sdague: I suspect your fix is going to break windows support a bit | 16:32 |
| sdague | dhellmann: why? | 16:32 |
| dhellmann | oh, maybe not, I see the _sighup_supported() call there now | 16:33 |
| *** rongze has quit IRC | 16:33 | |
| sdague | yeh | 16:33 |
| sdague | all I did was add another and False | 16:33 |
| sdague | which someone can replace with real daemon checking later | 16:33 |
| dhellmann | sdague: yeah, I was reading too quickly | 16:34 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/devstack-gate: WIP: test timout #2 https://review.openstack.org/62784 | 16:34 |
| sdague | I was hoping that would be uncontroversial enough to merge fast | 16:35 |
| sdague | this whole unstack not working.... is pretty annoying :) | 16:35 |
| dhellmann | sdague: yeah, this looks good, I'm just reading one more time carefully | 16:35 |
| dhellmann | sdague: of course, jenkins didn't like it | 16:35 |
| fungi | mordred: did your recent install_puppet.sh patches perchance fix bug 1218696? | 16:37 |
| uvirtbot | Launchpad bug 1218696 in openstack-ci "new puppetmaster ends up with broken pip provider" [Low,Triaged] https://launchpad.net/bugs/1218696 | 16:37 |
| jeblair | fungi: my guess would be yes | 16:39 |
| sdague | dhellmann: yeh, there were tests for the sighup behavior apparently | 16:40 |
| sdague | I might get back to it today | 16:40 |
| dhellmann | sdague: I'll try to pick it up in a bit if you can't get back to it | 16:40 |
| *** jlibosva has quit IRC | 16:41 | |
| *** talluri_ has quit IRC | 16:43 | |
| *** coolsvap has joined #openstack-infra | 16:45 | |
| *** rwsu has quit IRC | 16:47 | |
| ttx | annegentle_: done | 16:47 |
| annegentle_ | ttx: wow look at that flatline :) thank you | 16:48 |
| ttx | annegentle_: well, it's only counting since the last &0 minutes, so yes, pretty flat :) | 16:48 |
| ttx | 10* | 16:48 |
| openstackgerrit | A change was merged to openstack-infra/releasestatus: Add support for "Not" blueprint priority https://review.openstack.org/63375 | 16:49 |
| coolsvap | samalba: hi | 16:49 |
| annegentle_ | ttx: heh | 16:50 |
| annegentle_ | ttx: off all of next week? I'll be out til new year's | 16:52 |
| *** gyee_ has joined #openstack-infra | 16:53 | |
| ttx | annegentle_: I'm mostly out for the next two weeks | 16:53 |
| jeblair | fungi: the bare-precise image in iad is 14 hours old, i think that's after the change to performance merged, so i think we can merge the next one. agree? | 16:55 |
| fungi | jeblair: i think nodepool image 2128 (rax-iad,bare-precise) may have been built on a non-performance flavor, since i didn't notice puppet agent had been stopped on nodepool until close to when the daily image rebuilds started | 16:55 |
| fungi | i'm digging in the logs and db now to see whether i can confirm | 16:55 |
| jeblair | fungi: oh, ok | 16:55 |
| fungi | otherwise i'd have approved that next patch already | 16:56 |
| *** praneshp has quit IRC | 16:56 | |
| jeblair | fungi: sorry that was probably my fault; i stopped them (nodepool and zuul) when i was trying to figure out why puppetmaster was broken; i thought i restarted them but i guess not. | 16:56 |
| jeblair | fungi: i just restarted puppet on zuul | 16:57 |
| fungi | awesome--i hadn't noticed that one yet, so thanks! | 16:57 |
| *** smarcet has quit IRC | 16:59 | |
| annegentle_ | fungi: got a minute? | 16:59 |
| *** Ajaeger has joined #openstack-infra | 16:59 | |
| *** SergeyLukjanov is now known as _SergeyLukjanov | 17:00 | |
| annegentle_ | fungi: I'm talking through a good set of core approvers for the openstack-doc-tools repo with Andreas (ajaeger) and wanted to talk through with infra as well. | 17:00 |
| jeblair | fungi: in other news, the bulk of the devstack/openstack/tox/tempest processes are in the timeout process group; perhaps they get reparented or lose their controlling tty or something that causes timeout to miss them (certainly that's the case in the ps after timeout kills) | 17:00 |
| fungi | looks like nodepool.yaml was modified at 02:20 and the image build started at 02:16. missed it by 4 minutes... no nodes using that image though--should i just delete it? | 17:00 |
| annegentle_ | looks like clarkb jeblair mordred might have ideas too | 17:00 |
| *** _SergeyLukjanov has quit IRC | 17:00 | |
| jeblair | fungi: at any rate, i think i'll try having the script write the pgrp and then kill the group (again?) after the timeout | 17:01 |
| fungi | annegentle_: in the docs channel or on the ml? | 17:01 |
| jeblair | annegentle_: hi! | 17:01 |
| Ajaeger | Hi! | 17:01 |
| Ajaeger | jeblair, fungi: This is the review https://review.openstack.org/#/c/62984/ | 17:01 |
| Ajaeger | (that annegentle_ is talking about) | 17:01 |
| fungi | jeblair: sounds reasonable. it was hard to tell from the previous patchset what pgroup they ended up in | 17:02 |
| fungi | Ajaeger: annegentle_: i'll have a look | 17:02 |
| jeblair | Ajaeger, annegentle_: ah yeah, tagging -- there's no review or "undo" for pushing a tag, so we usually make a small group of people who can push a tag... | 17:02 |
| annegentle_ | so I haven't pushed tags yet... but I like the idea that the doc ptl would be the decision maker. | 17:02 |
| jeblair | Ajaeger, annegentle_: 1 sec and i'll get docs | 17:02 |
| annegentle_ | would also help with the clouddocs-plugin release decisions, I think | 17:02 |
| annegentle_ | but does the doc-ptl have to push the tag? | 17:02 |
| zaro | good morning | 17:03 |
| fungi | annegentle_: right. for the client libraries and other things which are not following the integrated release, it's usually the ptl and sometimes one or two release delegates for when the ptl wants a break | 17:03 |
| jeblair | annegentle_: https://wiki.openstack.org/wiki/GerritJenkinsGit#Tagging_a_Release | 17:03 |
| annegentle_ | fungi: ok that helps for more context | 17:03 |
| fungi | annegentle_: i meant to be clearer on that in the -infra ml thread we had going with Ajaeger but i was probably too vague about it | 17:04 |
| fungi | so sorry about that | 17:04 |
| annegentle_ | jeblair: I always have ttx do that for me | 17:04 |
| jeblair | annegentle_: yeah, he'll still do it for integrated projects, but since this one won't be part of the integrated release, you (or your delegate) get to! :) | 17:05 |
| annegentle_ | jeblair: well yee haw | 17:05 |
| jeblair | annegentle_: (so really you probably won't have a milestone-proposed branch like in the instructions, you'll probably just use master) | 17:05 |
| annegentle_ | jeblair: I could see using a "snapshot" which is what we're sort of used to for the maven plugin | 17:05 |
| fungi | annegentle_: it's not "hard" but if it's something you don't do frequently it's easy to forget a step or a command-line option. increasing the number of people doing that spreads out the tasks decreasing the frequency with which any one person does it, making forgetting much more likely | 17:05 |
| jeblair | annegentle_: i wouldn't recommend applying maven build practices to this -- the more it's like a standard pbr-style python library the better | 17:06 |
| annegentle_ | jeblair: ok, noted | 17:06 |
| annegentle_ | jeblair: fungi: is there a case where a "common library" ptl is appointed to have this ability? is that the case for oslo for example? | 17:07 |
| jeblair | annegentle_: i expect you'll use major.minor.micro versions, and you can just tag as many minor or micro versions as you want, and they'll get released, and then the doc builds can depend on those versions as you go. | 17:07 |
| Ajaeger | jeblair: My goal is to make openstack-doc-tools like a standard library - but it's my first time doing it, so I learn by doing ;) | 17:07 |
| jeblair | Ajaeger: cool :) | 17:07 |
| fungi | annegentle_: i think the oslo discussion around that ended up with the idea that the oslo ptl tapped different release delegates for the different libraries, but i haven't followed up to see whether that came to fruition | 17:08 |
| zaro | so what's the largest jenkins console log we have so far? | 17:08 |
| Ajaeger | jeblair: would we need to require the new version everywhere? Or can jenkins automatically use the latest release one? | 17:08 |
| annegentle_ | fungi: ok | 17:09 |
| jeblair | zaro: istr tempest console logs are about 25k lines | 17:09 |
| *** boris-42 has quit IRC | 17:09 | |
| *** hashar has quit IRC | 17:09 | |
| dhellmann | annegentle_, fungi : I think all of oslo-core can tag releases, but I'd have to go look again to verify that. | 17:09 |
| fungi | Ajaeger: it depends on how you wind up using it, but if the tests used tox you could just list that as a dep in the tox.ini and it would pip install -U the latest released version into the virtualenv it uses | 17:09 |
| jeblair | fungi, Ajaeger: right, if you list it without a version cap, it will use the latest one, or you can cap the version if you run into an incompatability or something. | 17:10 |
| fungi | and really, tox can be a very lightweight wrapper around the test script, so it's not that hard to add it into the mix | 17:11 |
| Ajaeger | dhellmann: you're right, for oslo, ever oslo-core can tag | 17:11 |
| annegentle_ | knowing what I know, I'm willing to tag releases | 17:11 |
| annegentle_ | for doc-tools | 17:12 |
| Ajaeger | fungi, jeblair: Ok, will look into that later. Thanks | 17:12 |
| Ajaeger | So, that means creating an acl for doc-ptl with annegentle_ in it, correct? | 17:12 |
| annegentle_ | and I don't think there's a -core that would otherwise do it, unless oslo-core was willing to also understand doc tool needs? | 17:12 |
| annegentle_ | Ajaeger: sounds like it to me | 17:12 |
| jeblair | annegentle_: make sure you have a gpg key set up too, btw (it's mentioned in the instructions; but just calling out that we want signed tags) | 17:13 |
| annegentle_ | jeblair: ah I should've gone to the signing party in hk | 17:13 |
| Ajaeger | annegentle_: Consider adding one fallback as release dude in case you're on vacation ;) | 17:13 |
| annegentle_ | Ajaeger: seriously | 17:13 |
| jeblair | annegentle_: we can get it signed later | 17:14 |
| annegentle_ | jeblair: okay | 17:14 |
| jeblair | fungi has a nice link to set up a key if you don't have one | 17:14 |
| annegentle_ | fungi: sure I'll take it | 17:15 |
| *** ruhe has quit IRC | 17:15 | |
| zaro | jeblair: thanks. that's actually a lot smaller than i expected. | 17:16 |
| *** coolsvap has quit IRC | 17:17 | |
| portante | hello folks, has anybody seen the "localhost" network adapter not come up in a check job? | 17:17 |
| portante | http://logs.openstack.org/28/51228/18/check/check-grenade-dsvm/c6e61ee/console.html | 17:17 |
| Ajaeger | jeblair, fungi : Do you need some formal request to create the ACL? And what's the name - so that I can adjust the patch. | 17:17 |
| *** derekh has quit IRC | 17:20 | |
| *** ruhe has joined #openstack-infra | 17:21 | |
| fungi | Ajaeger: openstack-doc-ptl would probably be fine, just use that instead of openstack-doc-core in the openstack-doc-tools.config [access "refs/tags/*"] section | 17:21 |
| *** dstanek has quit IRC | 17:22 | |
| Ajaeger | fungi: will update the patch now, thanks | 17:22 |
| fungi | annegentle_: https://wiki.openstack.org/wiki/OpenPGP_Web_of_Trust | 17:22 |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Further setup of openstack-doc-tools https://review.openstack.org/62984 | 17:23 |
| Ajaeger | jeblair, fungi: That's the updated patch ^ | 17:24 |
| *** ciypro has quit IRC | 17:25 | |
| portante | anyone? ... anyone? ... Bueller? ... Bueller? | 17:27 |
| fungi | portante: looking at the log, just trying to do too many things at once | 17:28 |
| *** coolsvap has joined #openstack-infra | 17:29 | |
| fungi | portante: where do you see the loopback interface not being up? the console log says it's up right there... something happened to abruptly terminate the job during setup_host though, so hopefully more detail in the other logs as to why | 17:30 |
| *** praneshp has joined #openstack-infra | 17:30 | |
| fungi | portante: the setup log makes it look like something happened during account creation | 17:31 |
| fungi | portante: actually, earliest signs of failure there are "mv: cannot stat `/home/jenkins/cache/pip/*': No such file or directory" | 17:32 |
| fungi | so to the workspace setup log, batman | 17:32 |
| fungi | error: Failed connect to zuul.openstack.org:80; Connection timed out while accessing http://zuul.openstack.org/p/openstack/swift/info/refs | 17:33 |
| *** boris-42 has joined #openstack-infra | 17:33 | |
| *** boris-42 has quit IRC | 17:33 | |
| fungi | looks like that machine may have met with some brief network connectivity issues | 17:33 |
| jeblair | fungi: yeah, there's a gap in the cacti monitoring | 17:34 |
| jeblair | for zuul | 17:34 |
| fungi | pow | 17:34 |
| fungi | so there is | 17:34 |
| * fungi checks rackspace status page | 17:34 | |
| fungi | doesn't look like they reported anything major | 17:34 |
| fungi | oh, wait, this log was from last night | 17:35 |
| fungi | https://status.rackspace.com/index/viewincidents?group=19&start=1387429200 | 17:35 |
| *** gyee_ has quit IRC | 17:35 | |
| jeblair | they really need to switch to utc | 17:36 |
| fungi | "...impacting some newly created performance servers..." so that could be iad (which is where that slave was) | 17:36 |
| *** Abhishek_ has joined #openstack-infra | 17:37 | |
| fungi | though still doesn't explain connectivity problems between cacti and zuul | 17:37 |
| jeblair | fungi: well, the cacti gap is later, so that probably wasn't it. | 17:37 |
| fungi | ahh, yeah | 17:37 |
| jeblair | fungi: there was also a moderate load spike on zuul around that time; could be cause or symptom. | 17:37 |
| fungi | oh, and i'm reading their notofocation log in the wrong order... "confirmed resolved as of approximately 23:36 CST on December 18th" | 17:38 |
| fungi | notification | 17:38 |
| jeblair | 5min load average was 16.6 at that time. | 17:38 |
| jeblair | we don't usually see problems until it's much higher | 17:39 |
| fungi | i'm still leaning toward "random network connectivity glitch affecting that slave, yay cloud" | 17:39 |
| jeblair | fungi: ++ | 17:39 |
| jeblair | portante: ^ | 17:40 |
| *** afazekas_ has quit IRC | 17:40 | |
| fungi | because it was communicating fine with zuul immediately prior to that error, then sudden connection timeout on one git operation | 17:40 |
| fungi | but we should keep an eye out for any similar examples--maybe will allow us to build a clearer picture of the incident if it's not a completely isolated one | 17:41 |
| *** krotscheck has joined #openstack-infra | 17:42 | |
| *** ruhe has quit IRC | 17:42 | |
| *** nati_ueno has joined #openstack-infra | 17:47 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/devstack-gate: WIP: test timout #2 https://review.openstack.org/62784 | 17:48 |
| *** yidclare has joined #openstack-infra | 17:48 | |
| *** dstanek has joined #openstack-infra | 17:48 | |
| fungi | jeblair: i confirmed there are still no nodes using the iad bare image, so i've deleted it and launched a new image-update for rax-iad bare-precise | 17:49 |
| fungi | not sure if you saw my earlier comment, but the periodic update missed the config change by 4 minutes | 17:50 |
| clarkb | morning | 17:51 |
| jeblair | fungi: ah, i missed that, sorry | 17:51 |
| jeblair | fungi: cool, thanks | 17:51 |
| *** dstanek has quit IRC | 17:53 | |
| portante | fungi: I was looking at the "lo: ... state UNKNOWN" line | 17:55 |
| portante | should that not say state UP | 17:55 |
| portante | ? | 17:55 |
| *** smarcet has joined #openstack-infra | 17:57 | |
| *** russellb is now known as rustlebee | 17:58 | |
| fungi | portante: does yours? http://paste.openstack.org/show/55722/ http://logs.openstack.org/83/62983/3/check/check-dg-tempest-dsvm-full/16e3ed5/console.html#_2013-12-18_22_45_25_229 et cetera... i believe link detection is irrelevant to the loopback | 17:59 |
| jeblair | fungi, clarkb: shut down 2 more logstash worker nodes (only 9-16 A and B are active now) | 18:00 |
| clarkb | jeblair: ok | 18:00 |
| clarkb | I am going to be messing around with groups on review-dev | 18:00 |
| fungi | awesome--curious to see what that does to the load on the remaining ones | 18:00 |
| clarkb | also zaro just mentioned in passing as he arrived that he thinks he has a fix for the scp plugin | 18:01 |
| *** markmcclain has joined #openstack-infra | 18:01 | |
| clarkb | jeblair: fungi: looking at the graphs for eth0 is also fun | 18:01 |
| fungi | yay! | 18:01 |
| jeblair | fungi, clarkb: i'm not certain it's completely representative, but over the past day, load average has peaked at 1.5; so i'm thinking we can have at least 4 processes / node | 18:01 |
| *** krotscheck has quit IRC | 18:01 | |
| fungi | jeblair: these are 4x vcpu servers, right? | 18:02 |
| jeblair | fungi: yes | 18:02 |
| fungi | seems likely to be a good sweet spot then | 18:02 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Increase logstash workers/host to 4 https://review.openstack.org/63474 | 18:02 |
| *** shardy is now known as shardy_afk | 18:02 | |
| jeblair | clarkb: i'm still not ready to really evaluate this (still more focused on getting it running); but this makes me optimistic: | 18:05 |
| jeblair | clarkb: http://logstash.openstack.org/#eyJzZWFyY2giOiJlcnJvcl9wcjpbLTEwMDAuMCBUTyAtODAuMF0iLCJmaWVsZHMiOlsiZXJyb3JfcHIiLCJtZXNzYWdlIl0sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiMzYwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjEzODc1NjI2NzIwNzZ9 | 18:05 |
| jeblair | (it's identifying tracebacks as log lines of interest) | 18:06 |
| clarkb | jeblair: nice | 18:06 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/devstack-gate: WIP: test timout #2 https://review.openstack.org/62784 | 18:09 |
| *** krotscheck has joined #openstack-infra | 18:10 | |
| *** Abhishek_ has quit IRC | 18:11 | |
| openstackgerrit | A change was merged to openstack-infra/config: Fix/improve projects_yaml_alphabetized script https://review.openstack.org/63353 | 18:12 |
| *** dhellmann is now known as dhellmann_ | 18:13 | |
| *** krotscheck has quit IRC | 18:14 | |
| * coolsvap is away: Away!!! | 18:14 | |
| *** krotscheck has joined #openstack-infra | 18:14 | |
| *** jog0 is now known as flashgordon | 18:14 | |
| fungi | coolsvap: please disable your away message | 18:15 |
| *** vkozhukalov has joined #openstack-infra | 18:15 | |
| *** yolanda_ has quit IRC | 18:17 | |
| clarkb | in testing by hand on review-dev group UUIDs seem to show up instantly | 18:18 |
| clarkb | so well under the 10 second timeout in jeepyb | 18:18 |
| *** harlowja_away is now known as harlowja | 18:19 | |
| *** EmilienM has quit IRC | 18:19 | |
| *** EmilienM has joined #openstack-infra | 18:20 | |
| *** yolanda_ has joined #openstack-infra | 18:21 | |
| openstackgerrit | A change was merged to openstack-infra/config: Switch to using IAD performance nodes https://review.openstack.org/63243 | 18:23 |
| fungi | as soon as i see bare nodes appearing in iad from that ^ i'll start an image-update for ord | 18:23 |
| jeblair | fungi: the 3rd change needs to merge before doing the ord image-update | 18:23 |
| fungi | jeblair: is the commit message on 63244 wrong then? | 18:24 |
| * clarkb blames his bad python mysqldb code now | 18:24 | |
| fungi | jeblair: i guess so, now that i compare it to the change and think about it a little more | 18:25 |
| fungi | in the commit message i think you must have meant "iad" instead of "ord" | 18:25 |
| jeblair | fungi: yep, sorry. | 18:26 |
| fungi | no worries. confusion resolved | 18:26 |
| fungi | and plan corrected accordingly | 18:26 |
| *** reed has joined #openstack-infra | 18:28 | |
| *** dstanek has joined #openstack-infra | 18:29 | |
| *** enikanorov__ has joined #openstack-infra | 18:29 | |
| *** enikanorov has quit IRC | 18:32 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/devstack-gate: WIP: test timout #2 https://review.openstack.org/62784 | 18:34 |
| *** SergeyLukjanov has joined #openstack-infra | 18:34 | |
| jeblair | every time i do that, 21 servers die. | 18:34 |
| fungi | i felt a great disturbance in the cloud, as if dozens of servers suddenly cried out in terror... and were suddenly silenced | 18:36 |
| reed | the post from enovance engineer about the port to Python3 seem weird to me... isn't there already a port effort ongoing? | 18:36 |
| fungi | reed: i got the impression he was unaware there were lots of openstackers already working on that goal | 18:36 |
| reed | fungi, I thought so too | 18:37 |
| fungi | but it could just be the way it's written. some people involved in the current effort do work at enovance | 18:37 |
| coolsvap | fungi: sorry for that | 18:37 |
| reed | too bad the blog doesn't allow comments | 18:38 |
| fungi | coolsvap: no problem. i figured it was on by accident. if i accidentally set my /away to echo in channel i'd want someone to tell me too ;) | 18:38 |
| coolsvap | fungi: sure that helps! I was just wondering in the channel for the whole day | 18:39 |
| SpamapS | Hm. I run this pypi-mirror thing and I swear it is _always_ compiling numpy | 18:39 |
| coolsvap | I am setting up the ci environment with devstack, and getting some issues with setting up the lxc docker driver | 18:39 |
| coolsvap | has anyone done this already? | 18:39 |
| fungi | coolsvap: i saw your discussion with lifeless earlier while most of us were still asleep | 18:40 |
| fungi | coolsvap: any new questions? | 18:40 |
| fungi | SpamapS: given the effort in compiling numpy, it should account for a disproportionately large percent of the run time | 18:40 |
| SpamapS | fungi: but.. numpy hasn't been updated in how long? so.. ?? | 18:41 |
| fungi | SpamapS: hopefully the wheels mirroring/caching work under way for pypi-mirror will improve that substantially | 18:41 |
| coolsvap | yes, I enabled the lxc driver, but getting some issues, even devstack does not getting setup | 18:41 |
| jd__ | reed: the author of that blog post knows about the ongoing effort, but he is still a bit new to OpenStack so don't blame him :) | 18:41 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/config: WIP: Explicitly document requirements for 3rd party testing https://review.openstack.org/63478 | 18:41 |
| reed | jd__, I don't blame him at all, i'd like to avoid that the post gets google juice and people think that the port to Py3 is not an agreed fact | 18:42 |
| fungi | SpamapS: we explicitly disable the egg cache in pypi-mirror because easy_install does dangerous things with it causing desired versions of things to get silently skipped (as we unfortunately discovered a while back) | 18:42 |
| jd__ | reed: sure, I get what you mean | 18:42 |
| SpamapS | fungi: _oh_ | 18:42 |
| SpamapS | fungi: so we have to rebuild everything every time ? Ok making more sense now. | 18:43 |
| reed | jd__, adding a comment would go a long way in that direction... too bad enovance blog doesn't allow that | 18:43 |
| jd__ | chmouel: ^^ | 18:43 |
| coolsvap | fungi: I checked the log output and I am hitting a in progress bug, so i was thinking i will use the libirt driver itself, how many vms will be required to set it up? | 18:43 |
| fungi | SpamapS: until wheels gets integrated more completely in pypi-mirror, yes. easy_install was just willing to reuse a locally-existing egg in place of a declared version spec, even if the version of the egg didn't fit that spec | 18:44 |
| fungi | coolsvap: how much of our ci toolchain are you attempting to duplicate on one machine, and at what scale (in other words, what bits do you need)? | 18:45 |
| SpamapS | fungi: oh thats just nutty | 18:45 |
| SpamapS | fungi: so wheels is not yet there? | 18:45 |
| *** krotscheck has quit IRC | 18:45 | |
| coolsvap | fungi: I am working on a poc to use zuul for solum git integration, i want to explore zuul | 18:46 |
| fungi | SpamapS: i think the work is mostly in place now. we had to revert some things a couple weeks back, particularly the per-distro stuff in select-mirror, i think because the wheels weren't being put in the right directories by pypi-mirror | 18:47 |
| fungi | SpamapS: i'd unfortunately need to go digging in the channel log for when review 60042 got proposed/approved for relevant context now (too many beers ago) | 18:48 |
| fungi | SpamapS: but others here may recall particular details | 18:48 |
| fungi | coolsvap: okay, so you need a zuul, and something to trigger it (probably a gerrit), and something to run things for it (probably a jenkins master) | 18:49 |
| *** dstanek has quit IRC | 18:49 | |
| coolsvap | fungi: yes | 18:49 |
| clarkb | fyi I have just done surgery on the jeepyb install on review-dev (puppet isn't running there and I needed latest jeepyb) | 18:50 |
| fungi | coolsvap: it *might* be possible to do a poc all in one system context (install zuul, jenkins and gerrit together in one vm) | 18:50 |
| *** rossella_s_ has joined #openstack-infra | 18:50 | |
| fungi | coolsvap: but that may run into issues with puppet configs and need some puppet changes to work correctly | 18:51 |
| *** dstanek has joined #openstack-infra | 18:51 | |
| fungi | coolsvap: so the lxc suggestion makes sense... three dedicated contexts, one for each of those things, with appropriate connectivity between them | 18:51 |
| fungi | coolsvap: using a devstack install to manage those contexts seems like a bit of overkill to me though | 18:52 |
| fungi | but others may disagree | 18:52 |
| fungi | clarkb: okay. we can re-enable puppet safely on it if we merge the gerrit upgrade test revert change | 18:52 |
| jeblair | i +2d it | 18:53 |
| fungi | i was pleasantly surprised that i was able to roll that upgrade back without having to do anything untoward | 18:53 |
| coolsvap | fungi: okay, just a bit more curious, how many vms will be required to setup single ci toolchain with all components? | 18:53 |
| SpamapS | fungi: I'll look in the git log | 18:53 |
| *** rossella_s has quit IRC | 18:54 | |
| *** rossella_s_ is now known as rossella_s | 18:54 | |
| SpamapS | fungi: master hasn't seen much activity mentioning wheels since Nov 16 | 18:54 |
| fungi | SpamapS: the reason i suggested the channel log is that the git log wasn't getting me the details i'd hoped for, but maybe you'll get lucky | 18:54 |
| fungi | coolsvap: so, to simplify, you could probably get away with 3... zuul, jenkins and gerrit. the jenkins master can actually run jobs locally so you don't really need slaves, pool management and those other fun things | 18:55 |
| SpamapS | fungi: ah :) | 18:56 |
| SpamapS | fungi: well I was on an older commit, so I've just pulled latest tip and I'll see if there is less building there :) | 18:56 |
| lifeless | fungi: but at that point you're really not running 'the openstack infrastructure' | 18:56 |
| lifeless | fungi: you're now running 'jenkins with zuul and gerrit' | 18:57 |
| fungi | lifeless: well, it's a core representation of that part of the workflow. adding nodepool, devstack slaves and a cloud provider and trying to cram all of that onto one laptop might be a recipe for fail | 18:57 |
| lifeless | fungi: sure; point I'm trying to get at is that the simulation requirements haven't been elucidated | 18:58 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/config: WIP: Explicitly document requirements for 3rd party testing https://review.openstack.org/63478 | 18:58 |
| lifeless | fungi: so we can at best be vague | 18:58 |
| fungi | not that it's not worth trying, but if the question is "how small can i go to test our zuul" | 18:58 |
| fungi | er, test out | 18:58 |
| fungi | (which was the most recent problem statement) | 18:59 |
| *** sarob_ has joined #openstack-infra | 19:01 | |
| fungi | coolsvap: so anyway, if you just want a working zuul you can exercise with code review and job running, those three components. if you want to "explore zuul's capabilities in the context of openstack's entire ci toolchain" then at least half a dozen, and that's likely a lowball estimate | 19:01 |
| *** ruhe has joined #openstack-infra | 19:01 | |
| fungi | also worth noting, though we haven't really tried it out in context yet, jhesketh__'s http://git.openstack.org/cgit/stackforge/turbo-hipster might let you eliminate the jenkins portion as well | 19:02 |
| *** jcooley_ has quit IRC | 19:03 | |
| fungi | (and has an awesome name) | 19:03 |
| *** yolanda_ has quit IRC | 19:04 | |
| *** ruhe has quit IRC | 19:05 | |
| jeblair | today's xkcd is apropos to my latest bash error (a missing $). :) | 19:05 |
| coolsvap | fungi: okay, i will give it a try to both turbo-hipster and 3 node setup you mentioned | 19:05 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/devstack-gate: WIP: test timout #2 https://review.openstack.org/62784 | 19:06 |
| openstackgerrit | A change was merged to openstack-infra/config: Switch ORD bare-precise to performance https://review.openstack.org/63244 | 19:07 |
| fungi | jeblair: clarkb: we have bare precise nodes in iad now, so i'm proceeding with the image update once ^ that gets to nodepool.o.o | 19:07 |
| jeblair | fungi: ++ | 19:08 |
| clarkb | k | 19:08 |
| *** rwsu has joined #openstack-infra | 19:08 | |
| clarkb | I am getting to the bottom of this jeepyb weirdness | 19:08 |
| clarkb | if I reconnect the DB it works | 19:08 |
| clarkb | not sure using a new cursor isn't sufficient | 19:08 |
| fungi | clarkb: it will be a relief to have that figured out once and for all | 19:08 |
| clarkb | I may just have jeepyb start a new connection each time it needs to talk to the db | 19:09 |
| jeblair | clarkb: transaction? | 19:09 |
| clarkb | jeblair: maybe? it is just doing a select | 19:09 |
| jeblair | clarkb: yeah, but i wonder if your transaction is holding a lock on the table that the group insert is waiting for | 19:10 |
| *** praneshp has quit IRC | 19:10 | |
| clarkb | oh I see. So there is still possibly a race and I am inserting a log jam | 19:11 |
| sdague | ttx: I think your bug might be something wrong with rax connectivity | 19:12 |
| sdague | wget https://pypi.python.org/simple/olso.config/ | 19:12 |
| sdague | fails on my rax nodes | 19:12 |
| *** zhiyan has left #openstack-infra | 19:13 | |
| clarkb | jeblair: but the sleep is after cursor.close | 19:14 |
| fungi | wow... milestone. i think 63316 is the most complex puppet change i've ever written which didn't turn out to be subtly broken in some way once i actually tried it out | 19:14 |
| * pleia2 cheers | 19:14 | |
| flashgordon | mordred: https://bugs.launchpad.net/neutron/+bug/1262906 | 19:15 |
| clarkb | I am sure I am just misreading python dbapi stuff /me reads more | 19:15 |
| uvirtbot | Launchpad bug 1262906 in neutron "swift unable to start in gate-tempest-dsvm-neutron-large-ops" [Undecided,Confirmed] | 19:15 |
| jeblair | flashgordon: what do you need? | 19:16 |
| flashgordon | jeblair: anteaya thought it may be HP AZ2 related | 19:16 |
| *** UtahDave has joined #openstack-infra | 19:16 | |
| flashgordon | jeblair: so the question is, are there still problems with it | 19:17 |
| jeblair | i don't recall hearing about az2 problems. | 19:17 |
| *** praneshp has joined #openstack-infra | 19:17 | |
| jeblair | clarkb: ? | 19:17 |
| clarkb | jeblair: apparently a transaction is implicitly started when a cursor is used, but closing the cursor does not do a commit. you have to commit on the connection object | 19:19 |
| jeblair | flashgordon: there are a lot of those failures; i doubt they all happened on az2 | 19:20 |
| jeblair | clarkb, flashgordon: node provider would be a nice column in logstash... | 19:21 |
| clarkb | jeblair: jenkins doesn't have that info | 19:21 |
| clarkb | well it does sort of | 19:21 |
| fungi | clarkb: so the cursor could actually be triggering this issue, rather than helping to mitigate it | 19:21 |
| jeblair | fungi, clarkb: yep. i suspect the implied transaction is holding a read lock on the table. | 19:21 |
| clarkb | jeblair: I agree though. there just isn't a clean way to put that data into the stream from what jenkins knows | 19:21 |
| flashgordon | jeblair: you may be right, I am stumped on that bug | 19:22 |
| *** rossella_s has quit IRC | 19:22 | |
| flashgordon | I am most confused by why it only happens in neutron-large-ops | 19:23 |
| jeblair | clarkb: i think we could get it from the node name (you're right it's not clean but it should work; that's what nodepool does (but it gets the id at the end)) | 19:23 |
| flashgordon | the test doesn't even get to tempest | 19:23 |
| jeblair | flashgordon: so the unique thing there is the fake libvirt driver, yeah? | 19:23 |
| flashgordon | jeblair: yes | 19:24 |
| flashgordon | and we set neutron quotas to -1 | 19:24 |
| fungi | nodepool image-update rax-ord bare precise is running now (confirmed performance is specified for it in nodepool.yaml) | 19:24 |
| flashgordon | and use neutron | 19:24 |
| *** hogepodge has joined #openstack-infra | 19:25 | |
| jeblair | flashgordon: rsyslogd is stopped but never restarted | 19:28 |
| jeblair | http://logs.openstack.org/70/62770/4/check/gate-tempest-dsvm-neutron-large-ops/d5f6251/logs/syslog.txt.gz | 19:28 |
| flashgordon | jeblair: right | 19:28 |
| jeblair | perhaps it is misconfigured? or perhaps there is a race condition with rsyslog restarting, and that test loses the race. | 19:29 |
| flashgordon | but how did that happen, what caused it to stop | 19:29 |
| jeblair | i believe it's normally restarted to pick up a different config | 19:29 |
| flashgordon | how come it only happens in thsi test? | 19:29 |
| flashgordon | ohh | 19:29 |
| flashgordon | hmm | 19:29 |
| jeblair | flashgordon: so if it's a race, maybe it's "restart syslog; do something; start swift" and in this test 'do something' is very fast and restart syslog hasn't actually finished yet | 19:30 |
| flashgordon | jeblair: that makes sense to me | 19:30 |
| flashgordon | lib/swift and stask.sh both restart it | 19:31 |
| jeblair | flashgordon: huh, looking at console log, i see "start_swift" then "restart_service rsyslog" | 19:32 |
| *** dhellmann_ is now known as dhellmann | 19:32 | |
| flashgordon | jeblair: sounds right | 19:32 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/jeepyb: Use MySQLDB transactions properly. https://review.openstack.org/63490 | 19:32 |
| flashgordon | first step in start_swift is restart_service rsyslog | 19:32 |
| clarkb | I tested ^ on review-dev (granted more of a contrived test of group creation and UUID reading not a test of full project creation) so I am pretty confident in it | 19:33 |
| fungi | clarkb: thanks! | 19:34 |
| flashgordon | 2013-12-20 19:29:20 rsyslog stop/waiting | 19:35 |
| clarkb | sdague: we are discussing swift fails here | 19:35 |
| flashgordon | Dec 20 19:29:16 devstack-precise-hpcloud-az2-906911 rsyslogd: [origin software="rsyslogd" swVersion="5.8.6" x-pid="2679" x-info="http://www.rsyslog.com"] exiting on signal 15. | 19:35 |
| sdague | clarkb: ah cool | 19:35 |
| flashgordon | jeblair: so syslog died befire the restart | 19:35 |
| sdague | we can't configure swift to actually write files instead? | 19:36 |
| jeblair | flashgordon: which run are you looking at? | 19:36 |
| fungi | what sent a sigterm to syslog i wonder? | 19:36 |
| clarkb | sdague: you can, but then you lose data | 19:36 |
| sdague | clarkb: why? | 19:36 |
| clarkb | sdague: swift logging relies on syslog for timestamps and stuff | 19:37 |
| sdague | really? swift can't put timestamps in files? | 19:37 |
| fungi | swift seems to be insistent that they should be deployed with a logserver if you want the real log experience | 19:37 |
| clarkb | though I think timestamps were added at the end of the lines in theproxy logs | 19:37 |
| sdague | that seems silly | 19:37 |
| clarkb | sdague: well it didn't before | 19:37 |
| flashgordon | http://logs.openstack.org/68/62668/3/check/gate-tempest-dsvm-neutron-large-ops/193ace6/logs/syslog.txt.gz | 19:37 |
| clarkb | I think it may now but would need to double check, this is very new | 19:37 |
| notmyname | yes, precise timestamps are in the logs now, in part because of -infra's request for them | 19:38 |
| flashgordon | why doesn't syslog have millisecond logging | 19:39 |
| clarkb | notmyname: thanks | 19:39 |
| flashgordon | hmm console.html says nova migrations are running at the same time syslog crashes | 19:40 |
| *** hub_cap is now known as pub_tap | 19:40 | |
| jeblair | s/crashes/exits/ | 19:40 |
| fungi | yeah, sigterm is not something which "just happens" | 19:41 |
| flashgordon | ack | 19:41 |
| *** blamar has quit IRC | 19:44 | |
| *** dprince has quit IRC | 19:44 | |
| fungi | jeblair: are you relatively confident https://review.openstack.org/63238 will meet the overhead needs of whatever solution you arrive at for actually implementing the job shutdown in d-g? or should i hold off approving for now? | 19:44 |
| *** sarob_ has quit IRC | 19:45 | |
| jeblair | fungi: i am, aprv at will | 19:45 |
| *** sarob_ has joined #openstack-infra | 19:45 | |
| anteaya | pub_tap: bet you are writing great code today | 19:46 |
| *** smarcet has left #openstack-infra | 19:46 | |
| *** smarcet has joined #openstack-infra | 19:46 | |
| flashgordon | anteaya: http://xkcd.com/323/ | 19:46 |
| fungi | free software development is powered mostly by beer | 19:47 |
| fungi | (and coffee) | 19:47 |
| *** blamar has joined #openstack-infra | 19:48 | |
| flashgordon | free as in beer | 19:48 |
| pub_tap | anteaya: :D | 19:48 |
| jeblair | flashgordon: rsyslog normally reloads during nova migrations (by observation successful runs of other jobs; i have no idea why) | 19:48 |
| anteaya | flashgordon: :D | 19:48 |
| openstackgerrit | A change was merged to openstack-infra/config: Supply a timeout value to devstack-gate https://review.openstack.org/63238 | 19:48 |
| *** homeless has quit IRC | 19:48 | |
| jeblair | or at least, that's what the timestamps say | 19:48 |
| *** blamar has quit IRC | 19:49 | |
| flashgordon | jeblair: wow strange | 19:49 |
| anteaya | pub_tap: may you find the peak | 19:49 |
| *** blamar has joined #openstack-infra | 19:49 | |
| *** sarob_ has quit IRC | 19:50 | |
| pub_tap | and fall down whilst looking! | 19:50 |
| anteaya | ha ha ha | 19:52 |
| *** dstanek has quit IRC | 19:53 | |
| flashgordon | jeblair: a spot check confirms, this is getting stranger | 19:54 |
| jeblair | fungi: http://logs.openstack.org/84/62784/12/check/check-dg-tempest-dsvm-full/8b14c50/console.html#_2013-12-20_19_19_30_734 | 19:54 |
| jeblair | fungi: according to that, the timeout program is returning 137 after a timeout. it's supposed to return 124. | 19:55 |
| fungi | uhh | 19:55 |
| flashgordon | jeblair: this bug is getting stranger all the time | 19:55 |
| portante | fungi: yes, the "ip ad sh" output on my box also shows it as UNKNOWN, so that is not it | 19:56 |
| portante | is it worth a bug report regardless? | 19:56 |
| jeblair | fungi: oh, it seems to do that if you use -s 9 | 19:56 |
| jeblair | nice docs. :/ | 19:56 |
| fungi | jeblair: ahh. i was starting to wonder whether it was a bash builtin quirk or something | 19:57 |
| fungi | less mysterious then | 19:58 |
| jeblair | fungi: http://www.gnu.org/software/coreutils/manual/html_node/timeout-invocation.html | 19:58 |
| *** coolsvap has quit IRC | 19:58 | |
| fungi | portante: sure, it can't hurt, and then if we get any related occurrences it might help us figure out whether the timing was consistent | 19:58 |
| *** coolsvap has joined #openstack-infra | 19:58 | |
| jeblair | fungi: that's not even in my copy of 'info timeout' | 19:58 |
| portante | fungi: so this would be filed against openstack-ci? | 19:59 |
| *** eharney has quit IRC | 19:59 | |
| fungi | portante: yes, it's as close as anything to the right place i guess (since we have the ability to open a ticket with the cloud provider at least) | 19:59 |
| clarkb | I feel like there are three conversations going on here and detangling them is hard | 20:00 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/devstack-gate: WIP: test timout #2 https://review.openstack.org/62784 | 20:00 |
| *** eharney has joined #openstack-infra | 20:00 | |
| *** dstanek has joined #openstack-infra | 20:00 | |
| jeblair | clarkb: indeed i'm carrying on two. | 20:00 |
| fungi | portante: i'll just make a note in it and mark it incomplete so it's there if we get more reports | 20:00 |
| *** sarob_ has joined #openstack-infra | 20:00 | |
| jeblair | clarkb: with some of the same people. | 20:00 |
| clarkb | oh I see the address one is from way before | 20:00 |
| clarkb | then we have timeouts and neutron largeops. I am caught up now | 20:01 |
| flashgordon | clarkb: neutron large ops https://bugs.launchpad.net/neutron/+bug/1262906 | 20:01 |
| uvirtbot | Launchpad bug 1262906 in neutron "swift unable to start in gate-tempest-dsvm-neutron-large-ops" [Undecided,Confirmed] | 20:01 |
| fungi | clarkb: yeah there was a job which got a socket timeout trying to reach zuul yesterday, but it surfaced in workspace setup, so wasn't immediately obvious why the console log ended so abruptly | 20:01 |
| *** _david_ has joined #openstack-infra | 20:02 | |
| *** flaper87 is now known as flaper87|afk | 20:02 | |
| flashgordon | moving large ops discussion to qa | 20:03 |
| fungi | jeblair: so i guess the intent with the timeout utility's exit codes is that success with a particular signal should be indicated with 128+signal, they simply never implemented any others? | 20:03 |
| *** sarob_ has quit IRC | 20:04 | |
| clarkb | fungi: looks likeit | 20:05 |
| fungi | just odd that you can timeout with sigabrt for example, but it doesn't explicitly say that returns 134 when it happens | 20:05 |
| clarkb | fungi: does it return 134 ? | 20:06 |
| fungi | clarkb: i don't know. it seems like it should, but the fine manual doesn't really say one way or the othert | 20:07 |
| * clarkb checks | 20:07 | |
| fungi | i just find it amusing that there's some chance they devised a signal-to-exit-code mapping scheme and then applied it to only one signal | 20:08 |
| clarkb | fungi: doesn't seem to | 20:08 |
| fungi | pretty awesome in that case | 20:09 |
| portante | fungi: https://bugs.launchpad.net/openstack-ci/+bug/1263221 | 20:09 |
| uvirtbot | Launchpad bug 1263221 in openstack-ci "check job was unable to create stack user directory" [Undecided,New] | 20:09 |
| fungi | portante: thanks! | 20:09 |
| portante | could the devstack script error out when the stack user can't be created? | 20:09 |
| portante | it seems pretty fundamental ... | 20:09 |
| fungi | portante: it looks like that probably happened, but was a cascade failure stemming from the inability to clone that repo | 20:10 |
| *** thedodd has joined #openstack-infra | 20:10 | |
| portante | okay, thanks | 20:10 |
| fungi | er, inability to fetch a swift ref from zuul, i mean | 20:12 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/devstack-gate: WIP: test timout #2 https://review.openstack.org/62784 | 20:15 |
| *** boris-42 has joined #openstack-infra | 20:23 | |
| clarkb | zaro just gave me an scp hpi build to try. going to restart jenkins-dev in a moment to install that | 20:24 |
| *** metabro_ has quit IRC | 20:27 | |
| *** metabro has joined #openstack-infra | 20:27 | |
| clarkb | and it is installed. Going to find food before testing that more | 20:29 |
| *** ArxCruz has quit IRC | 20:29 | |
| *** yolanda_ has joined #openstack-infra | 20:30 | |
| jeblair | fungi: looks like a working timeout poc now: https://jenkins01.openstack.org/job/check-dg-tempest-dsvm-full/41/console | 20:31 |
| clarkb | cool | 20:37 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/devstack-gate: Timeout devstack/tempest internally https://review.openstack.org/63512 | 20:40 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/devstack-gate: Timeout devstack/tempest internally https://review.openstack.org/63512 | 20:40 |
| jeblair | clarkb, fungi, flashgordon: ^ that's ready for review | 20:41 |
| *** UtahDave has quit IRC | 20:41 | |
| *** UtahDave has joined #openstack-infra | 20:42 | |
| fungi | jeblair: did sudo kill -s 9 -${GATEPID} actually work? i've usually needed sudo kill -s 9 -- -${GATEPID} to keep it from being interpreted as a cli option | 20:43 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/devstack-gate: Remove strange duplication of setup_host and output https://review.openstack.org/62983 | 20:43 |
| *** yassine has quit IRC | 20:43 | |
| jeblair | fungi: it seemed to; at least, everything in the pgroup was dead or defunct in the last ps listing | 20:44 |
| fungi | okay, great. it could be an interactive vs script interpreter difference too | 20:45 |
| *** CaptTofu has joined #openstack-infra | 20:50 | |
| anteaya | clarkb: are logstash timestamps utc? | 20:52 |
| *** Protux has joined #openstack-infra | 20:54 | |
| *** jcooley_ has joined #openstack-infra | 20:58 | |
| fungi | jeblair: clarkb: mordred: we're starting to get into openid application deployment design discussions on https://review.openstack.org/63316 so just want to make sure that's on your radar | 21:02 |
| fungi | odd things like having two databases, configuration comingled with code and so on, so any input/opinions appreciated | 21:03 |
| * fungi is going out to dinner, but will return to continue in a little while | 21:04 | |
| *** smarcet has left #openstack-infra | 21:11 | |
| *** rwsu has quit IRC | 21:13 | |
| *** thedodd has quit IRC | 21:17 | |
| mfer | folks, i'm poking around the golang client. for internal reference i need to reference the full url to the repo. should i use the github location for stackforge or would that be bad form? | 21:18 |
| clarkb | anteaya: yes utc | 21:18 |
| clarkb | jeblair: ok | 21:19 |
| anteaya | clarkb: thanks | 21:20 |
| anteaya | thought so, but wanted to confirm before I ruled something out | 21:20 |
| *** krotscheck has joined #openstack-infra | 21:24 | |
| *** lcestari has quit IRC | 21:27 | |
| *** Ajaeger has quit IRC | 21:28 | |
| *** chandankumar_ has joined #openstack-infra | 21:31 | |
| *** rwsu has joined #openstack-infra | 21:33 | |
| *** thedodd has joined #openstack-infra | 21:40 | |
| *** thedodd has quit IRC | 21:40 | |
| *** CaptTofu has quit IRC | 21:40 | |
| *** mfer has quit IRC | 21:42 | |
| *** harlowja is now known as harlowja_away | 21:44 | |
| *** jcooley_ has quit IRC | 21:48 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/devstack-gate: Timeout devstack/tempest internally https://review.openstack.org/63512 | 21:49 |
| jeblair | oops. minutes, not seconds. | 21:49 |
| *** jcooley_ has joined #openstack-infra | 21:49 | |
| *** adalbas has quit IRC | 21:57 | |
| openstackgerrit | A change was merged to openstack-infra/config: Remove status.o.o vhost from static.o.o https://review.openstack.org/63234 | 21:58 |
| *** gothicmindfood has quit IRC | 21:58 | |
| *** eharney has quit IRC | 21:59 | |
| *** praneshp has quit IRC | 22:01 | |
| dims | jeblair, ping | 22:02 |
| *** praneshp has joined #openstack-infra | 22:02 | |
| *** hashar has joined #openstack-infra | 22:02 | |
| jeblair | dims: pong | 22:03 |
| dims | jeblair, in devstack-gate.yaml file, most of neutron jobs have timeout of 120 except from neutron-large-ops. is this by design? | 22:05 |
| jeblair | dims: i don't know; sdague and flashgordon are probably the best to ask about that | 22:06 |
| sdague | dims: yes, by design | 22:06 |
| *** melwitt has joined #openstack-infra | 22:06 | |
| sdague | large ops shouldn't take that long | 22:07 |
| openstackgerrit | Julien Danjou proposed a change to openstack-dev/pbr: package: read a specific Python version requirement file https://review.openstack.org/63236 | 22:08 |
| *** hashar has quit IRC | 22:09 | |
| dims | sdague, jeblair cool thx | 22:09 |
| clarkb | the scp file copy is tricky... | 22:17 |
| jeblair | flashgordon, sdague: i held a node with the swift/rsyslog error | 22:17 |
| *** dkliban has quit IRC | 22:17 | |
| flashgordon | jeblair: cool not even sure what to check for though | 22:19 |
| sdague | jeblair: unrelated - pypi mirror issues again? - https://jenkins01.openstack.org/job/gate-nova-docs/16266/console | 22:19 |
| jeblair | root 2582 0.0 0.0 17284 1212 ? Ss 22:13 0:00 rsyslogd -c5 | 22:19 |
| jeblair | root 2584 0.0 0.0 0 0 ? Z 22:13 0:00 [rsyslogd] <defunct> | 22:19 |
| flashgordon | dmesg may be useful?? | 22:19 |
| sdague | flashgordon: it wasn't in the other runs | 22:20 |
| *** mordred has quit IRC | 22:20 | |
| *** mordred has joined #openstack-infra | 22:20 | |
| sdague | we capture that | 22:20 |
| jeblair | [task switching to pypi mirror] | 22:20 |
| *** markmcclain has quit IRC | 22:20 | |
| *** praneshp has quit IRC | 22:21 | |
| *** SergeyLukjanov has quit IRC | 22:22 | |
| jeblair | sdague: i don't see anything systemically wrong, and it's not under particularly heavy load. | 22:23 |
| flashgordon | dims sdague: any ideas to check | 22:23 |
| *** harlowja_away is now known as harlowja | 22:23 | |
| flashgordon | jeblair: maybe look at rsyslog config? | 22:23 |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/config: Make cells job gating https://review.openstack.org/63537 | 22:24 |
| jeblair | i apparently held it just too late; nodepool was already deleting it :( | 22:24 |
| flashgordon | so the first case of the bug was in a tempest patch | 22:27 |
| clarkb | going to restart jenkins-dev against shortly to try another scp.hpi | 22:28 |
| *** sdake has quit IRC | 22:28 | |
| flashgordon | which means this snuck through the gate or is due to an external dep changing | 22:29 |
| flashgordon | do we have any good logs of pip mirror changes | 22:29 |
| flashgordon | http://logs.openstack.org/38/60738/5/gate/gate-tempest-dsvm-neutron-large-ops/2a6f31d/logs/pip-freeze.txt.gz | 22:30 |
| flashgordon | thats a lot of deps | 22:31 |
| *** julim has quit IRC | 22:35 | |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/config: Make cells job gating https://review.openstack.org/63537 | 22:40 |
| *** ArxCruz has joined #openstack-infra | 22:42 | |
| *** rnirmal has quit IRC | 22:44 | |
| *** ArxCruz has quit IRC | 22:46 | |
| *** dkliban has joined #openstack-infra | 22:47 | |
| jeblair | flashgordon: i spot checked a couple of successful hosts, and there is no zombie rsyslog process | 22:48 |
| jeblair | flashgordon: so the one from the failed host is probably related to the failure | 22:48 |
| flashgordon | jeblair: hmm | 22:49 |
| dims | flashgordon, can't find anything in the logs so far | 22:49 |
| flashgordon | so I am wondering what is causing the first rsyslog restart in the first palce | 22:49 |
| flashgordon | place | 22:49 |
| flashgordon | why does nova-manage db_sync do that | 22:50 |
| flashgordon | or rather why does rsyslog restart at the same time (not convinced there is causality yet) | 22:50 |
| dims | agree | 22:52 |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/config: zuul status: Collapse changes by default https://review.openstack.org/63019 | 22:53 |
| dims | flashgordon, look at this | 22:55 |
| dims | dims@dims-laptop:~/junk$ grep -e start_swift -e configure_swift devstacklog.txt.gz | 22:55 |
| dims | 2013-12-20 21:11:40 + configure_swift | 22:55 |
| dims | 2013-12-20 21:14:00 + start_swift | 22:55 |
| dims | dims@dims-laptop:~/junk$ grep 21:13:56 syslog.txt.gz | 22:55 |
| dims | Dec 20 21:13:56 devstack-precise-hpcloud-az2-907665 kernel: Kernel logging (proc) stopped. | 22:55 |
| dims | Dec 20 21:13:56 devstack-precise-hpcloud-az2-907665 rsyslogd: [origin software="rsyslogd" swVersion="5.8.6" x-pid="2602" x-info="http://www.rsyslog.com"] exiting on signal 15. | 22:55 |
| dims | there is a restart_service rsyslogd in start_swift as well | 22:55 |
| dims | gotta run. back later | 22:57 |
| flashgordon | dims: yeah but that restart is after the one that happens during nova-manage db_sync | 22:57 |
| flashgordon | o/ | 22:57 |
| jeblair | [i'm still working on catching another node] | 22:59 |
| *** dcramer_ has quit IRC | 23:01 | |
| *** david-lyle_ has joined #openstack-infra | 23:01 | |
| *** melwitt has quit IRC | 23:02 | |
| *** prad has quit IRC | 23:02 | |
| *** melwitt has joined #openstack-infra | 23:03 | |
| flashgordon | jeblair: thanks | 23:05 |
| *** prad has joined #openstack-infra | 23:06 | |
| *** prad has quit IRC | 23:06 | |
| flashgordon | can you get ahold of a rsyslog config file | 23:06 |
| *** yolanda_ has quit IRC | 23:07 | |
| *** david-lyle_ has quit IRC | 23:08 | |
| *** sarob_ has joined #openstack-infra | 23:09 | |
| *** praneshp has joined #openstack-infra | 23:10 | |
| flashgordon | or any sql logs | 23:11 |
| jeblair | flashgordon: that's what i'm working on. | 23:14 |
| flashgordon | jeblair: awesome | 23:14 |
| jeblair | have to find an affected node first | 23:14 |
| *** sarob_ has quit IRC | 23:14 | |
| flashgordon | I am just thinking out loud over here | 23:14 |
| flashgordon | well if we are just looking for why rsyslog restarts during nova-manage db_sync | 23:14 |
| flashgordon | thatis every run | 23:15 |
| *** vkozhukalov has quit IRC | 23:15 | |
| flashgordon | and that may give us some more insight into the bug | 23:15 |
| *** sarob has joined #openstack-infra | 23:18 | |
| jeblair | flashgordon: /etc/rsyslog.d http://paste.openstack.org/show/55759/ | 23:19 |
| flashgordon | hmm i dont see anything strange in there | 23:20 |
| flashgordon | sql logs? | 23:21 |
| sdague | so the only place that syslog is getting restarted is in lib/swift | 23:22 |
| sdague | at least from what I can see in reading relevant code | 23:22 |
| flashgordon | sdague: right | 23:22 |
| flashgordon | but look at the syslog output | 23:22 |
| flashgordon | and timestamps don't match up | 23:22 |
| sdague | where is a fail link? | 23:22 |
| sdague | url for logs please | 23:22 |
| *** CaptTofu has joined #openstack-infra | 23:23 | |
| jeblair | flashgordon: i think the sql thing is a red herring | 23:24 |
| flashgordon | jeblair: yeahh | 23:24 |
| flashgordon | sdague: http://logs.openstack.org/41/62641/2/check/gate-tempest-dsvm-neutron-large-ops/ad907d1/logs/syslog.txt.gz | 23:24 |
| flashgordon | working run | 23:24 |
| flashgordon | http://logs.openstack.org/41/62641/2/check/gate-tempest-dsvm-neutron-large-ops/ad907d1/console.html#_2013-12-20_09_33_54_050 | 23:25 |
| flashgordon | rsyslog timestamp is 4 seconds before | 23:25 |
| flashgordon | perhaps the timestamps just don't match | 23:25 |
| sdague | flashgordon: when did ceilometer start syslogin? | 23:27 |
| sdague | could this just be that syslog crash problem we saw before when it gets too much data | 23:27 |
| flashgordon | jeblair: can you check the upstart log forrsyslog | 23:27 |
| jeblair | flashgordon: no, because i don't have a failed host. | 23:28 |
| flashgordon | /var/log/upstart/rsyslog or something | 23:28 |
| flashgordon | jeblair: oh right | 23:28 |
| flashgordon | sdague:don't think so because ceilometer isn't running at that moment | 23:29 |
| jeblair | sdague: i briefly had a failed host and rsyslog was running, starting from around the expected time. there was also a zombie rsyslog process, which is not normal (not present on successful hosts) | 23:29 |
| sdague | flashgordon: if it sent a giant packet that killed it, it wouldn't have written | 23:29 |
| flashgordon | sdague: compare the bad syslog http://logs.openstack.org/60/61660/4/gate/gate-tempest-dsvm-neutron-large-ops/dd42154/logs/syslog.txt.gz | 23:30 |
| jeblair | flashgordon, sdague: i think i have one | 23:30 |
| flashgordon | with good http://logs.openstack.org/41/62641/2/check/gate-tempest-dsvm-neutron-large-ops/ad907d1/logs/syslog.txt.gz | 23:30 |
| jeblair | flashgordon, sdague: http://logs.openstack.org/60/61660/4/gate/gate-tempest-dsvm-neutron-large-ops/dd42154/ | 23:30 |
| * flashgordon feels like jeblair is going fishing | 23:31 | |
| jeblair | let me reel it in! | 23:31 |
| jeblair | syslog is _not_ running on this host | 23:31 |
| flashgordon | \o/ | 23:32 |
| jeblair | http://paste.openstack.org/show/55760/ | 23:32 |
| *** jhesketh has quit IRC | 23:32 | |
| flashgordon | WAT | 23:33 |
| *** jerryz has joined #openstack-infra | 23:34 | |
| *** rcleere has quit IRC | 23:36 | |
| sdague | so it's crashing for some reason, swift attempts to restart it, and it fails because there is already a pid file? | 23:36 |
| sdague | are these only on certain kinds of nodes? | 23:36 |
| flashgordon | what version of rsyslog is installed | 23:36 |
| jeblair | sdague: i don't think we've seen evidence it crashed | 23:36 |
| jeblair | Dec 20 23:16:15 devstack-precise-hpcloud-az2-908486 sudo: stack : TTY=unknown ; PWD=/opt/stack/new/devstack ; USER=root ; COMMAND=/usr/sbin/service rsyslog restart | 23:36 |
| sdague | Dec 20 23:16:15 devstack-precise-hpcloud-az2-908486 rsyslogd: [origin software="rsyslogd" swVersion="5.8.6" x-pid="2622" x-info="http://www.rsyslog.com"] exiting on signal 15. | 23:36 |
| sdague | ok, not crashed, someone sent it a TERM | 23:37 |
| jeblair | sdague: sigterm is not crashing | 23:37 |
| flashgordon | apt-cache show rsyslog | 23:37 |
| flashgordon | sdague: service dd | 23:37 |
| flashgordon | sdague: sudo /usr/sbin/service rsyslog restart | 23:37 |
| jeblair | yeah, so that's ^^ from auth.log, which shows a restart command synced with the sigterm | 23:37 |
| jerryz | fungi: ping | 23:37 |
| fungi | jerryz: what's up? | 23:38 |
| jeblair | ii rsyslog 5.8.6-1ubuntu8.6 reliable system and kernel logging daemon | 23:38 |
| sdague | so the devstack timestamps come from a filter right | 23:38 |
| sdague | so those could just be delayed | 23:38 |
| fungi | "reliable" seems ironic in the current context | 23:38 |
| sdague | because at :18 is the swift initiated restart | 23:38 |
| sdague | in the devstack log timestamps | 23:39 |
| jerryz | fungi: if i want to add a related project under stackforge, should i just submit a change on openstack-infra/config? Do i need some more authorization or just reviewed by infra guys? | 23:39 |
| flashgordon | http://changelogs.ubuntu.com/changelogs/pool/main/r/rsyslog/rsyslog_5.8.6-1ubuntu8.6/changelog | 23:39 |
| flashgordon | I wonder how to check when the latest rsyslog got into our images | 23:39 |
| fungi | jerryz: that's all you need | 23:39 |
| sdague | flashgordon: that change is too far back to be relevant I think | 23:40 |
| sdague | this was pretty recent | 23:40 |
| flashgordon | sdague: I agree with you, but don't know when we started using that version | 23:40 |
| *** hogepodge has quit IRC | 23:41 | |
| jeblair | flashgordon: they are rebuilt daily, and i'm not aware of a problem preventing their updates, so we should have gotten it within a day of it propogating to the archives | 23:41 |
| jerryz | fungi: once i feel confident adding tests, i just add them in jjb and zuul, but i need make them work with test commands like other openstack projects, it that correct? | 23:41 |
| flashgordon | jeblair: we use ubuntu mirrors? or others | 23:42 |
| flashgordon | anyway to spot check a image from the 17th or something | 23:42 |
| jeblair | flashgordon: this is az2 so it'll be an ubuntu mirror unless hpcloud mitm them. | 23:42 |
| jeblair | flashgordon: let me see if the image build logs help | 23:43 |
| jeblair | 2013-12-17 02:19:34,994 INFO nodepool.image.build.hpcloud-az2.devstack-precise: Preparing to replace rsyslog 5.8.6-1ubuntu8 (using .../rsyslog_5.8.6-1ubuntu8.6_amd64.deb) ... | 23:43 |
| flashgordon | oh well | 23:44 |
| fungi | jerryz: basically, yes (if you stick to running things the same way, then it's more likely to work without needing to justify additional resources) | 23:44 |
| flashgordon | that would haave been too easy | 23:44 |
| sdague | jerryz: not sure if you followed the list, but did get to the bottom of the of the unstack issue | 23:45 |
| jerryz | sdague: i didn't. what is it? | 23:45 |
| jeblair | flashgordon: 2013-12-08 is when that showed up everywhere | 23:45 |
| sdague | jerryz: https://review.openstack.org/#/c/63495/ | 23:46 |
| clarkb | we have tracked down that the file copy stuff is failing because the remote sftp subsystem disconnects when it receives data for a channel it doesn't know about | 23:48 |
| *** SumitNaiksatam has joined #openstack-infra | 23:48 | |
| *** boris-42 has quit IRC | 23:48 | |
| clarkb | not entirely sure why that is happening, but going to beef up logging and see if we can figure it out | 23:48 |
| flashgordon | jeblair: any other ideas? | 23:49 |
| SumitNaiksatam | just sent a request for a gerrit service account: #link http://lists.openstack.org/pipermail/openstack-infra/2013-December/000556.html | 23:50 |
| sdague | jeblair: we know /opt/stack/data/swift/logs/ looks good? | 23:51 |
| SumitNaiksatam | hopefully some might get a chance to take a look at the service account request ^^ | 23:51 |
| SumitNaiksatam | thanks in advance | 23:51 |
| jeblair | sdague: http://paste.openstack.org/show/55763/ | 23:52 |
| jeblair | sdague: i have not tried to start rsyslog yet (trying read-only investigations so far) | 23:52 |
| *** chandankumar_ has quit IRC | 23:53 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/devstack-gate: Collect list of installed packages at end of run https://review.openstack.org/63551 | 23:54 |
| jerryz | sdague: thanks for the heads up | 23:54 |
| flashgordon | so why only this job? | 23:55 |
| *** CaptTofu has quit IRC | 23:55 | |
| sdague | flashgordon: it only fails on the neutron large ops? | 23:55 |
| *** _david_ has quit IRC | 23:56 | |
| flashgordon | sdague: yup | 23:56 |
| flashgordon | but please don't take my word for it | 23:57 |
| flashgordon | as I don't belive it myself | 23:57 |
| jeblair | http://paste.openstack.org/show/55767/ | 23:57 |
| jeblair | sdague, flashgordon: seems worth noting that ovs interacts with syslog | 23:57 |
| jeblair | shall i start trying to restart rsyslogd now? | 23:58 |
| sdague | so it's also notable that we're running in SYSLOG=False in devstack... and swift still is going through syslog. There is actually a bunch of code in devstack to tune rsyslog not getting used | 23:59 |
| sdague | anyway, I'm getting called away | 23:59 |
| flashgordon | hmm | 23:59 |
| sdague | have a good weekend folks | 23:59 |
| flashgordon | jeblair: go for it | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!