| *** CaptTofu has quit IRC | 00:00 | |
| *** CaptTofu has joined #openstack-infra | 00:00 | |
| *** virmitio has quit IRC | 00:02 | |
| fungi | i am voiding the warranty on my ps3 so that i can clean the dust out of it. srsly sony? | 00:03 |
|---|---|---|
| clarkb | hahaha | 00:03 |
| clarkb | fungi: those machines are only supposed to last a couple years then ouy need to buy the new slim extra locked down edition | 00:04 |
| clarkb | flashgordon: so grenade libvirtd logs aren't parsing correctly but other jobs are. I am looking into it | 00:06 |
| flashgordon | clarkb: thanks! I wonder if its because the old devstack to new upgrade that messes up the config | 00:07 |
| clarkb | flashgordon: oh I know | 00:08 |
| clarkb | not all of our gearman workers have updated | 00:08 |
| clarkb | because yeah. let go go around and check that they are all happy | 00:08 |
| flashgordon | ^_^ | 00:08 |
| ktbenton | fungi: I already made the change and it seems to be okay | 00:12 |
| ktbenton | fungi: oh, but i think my id did change | 00:13 |
| ktbenton | fungi: my username stayed the same and it still has all of my ssh keys | 00:14 |
| *** mcriderman has quit IRC | 00:14 | |
| ktbenton | fungi: can you search by my email address to see if there is a duplicate account? | 00:14 |
| fungi | ktbenton: yes, easily. what's your preferred e-mail address in gerrit? | 00:15 |
| fungi | (well, easily is a mysql select query performed over a local socket on the server, but easily enough) | 00:15 |
| ktbenton | fungi: bla111 <at> gmail <at> com | 00:15 |
| fungi | checking | 00:15 |
| ktbenton | fungi: whoops | 00:15 |
| ktbenton | fungi: blak111 <at> gmail <dot> com | 00:16 |
| ktbenton | fungi: missed the k | 00:16 |
| fungi | ktbenton: i only see an account 7787 with that preferred e-mail address. i'll look for other signs of duplication though just to be certain | 00:17 |
| clarkb | flashgordon: I think we are ok now. I haven't seen anymore with the problem over the last 8 minutes. It just takes time to rollout everyhwere | 00:17 |
| clarkb | I should salt this stuff | 00:17 |
| ktbenton | fungi: ok. i think it did work. i just don't remember what the id previously showed so i want to be sure | 00:18 |
| ktbenton | fungi: thanks for your help | 00:18 |
| fungi | ktbenton: i think you're fine. gerrit says that id signed the icla on 2013-06-04 so doesn't seem to have any duplication | 00:19 |
| fungi | happy hacking! | 00:19 |
| flashgordon | clarkb: nice, this should help us better grasp how bad libvirt is right now | 00:19 |
| fungi | flashgordon: hint... "very" ;) | 00:20 |
| fungi | we're so far behind on libvirtd releases we use, yet until we can get the new libvirtd working... *sigh* | 00:20 |
| flashgordon | fungi: yeah ... soon we can quantify very | 00:20 |
| clarkb | flashgordon: I can quantify it. latest libvirt doesn't work 100% of the time for tempest :) | 00:21 |
| clarkb | danpb found the bug according to his notes in the launchpad bug and is working it | 00:21 |
| clarkb | but it literally does not work at all | 00:21 |
| flashgordon | clarkb: and old only sorta works | 00:21 |
| flashgordon | (0.9.8) | 00:21 |
| clarkb | in any case we now have greande logs and libvirt logs, so I expect good things :) | 00:22 |
| clarkb | flashgordon: speaking of should we just do the filename restriction in one go right now for e-r? | 00:22 |
| clarkb | I am happy to review that if you propose it, but I am fading fast | 00:22 |
| flashgordon | clarkb: sure | 00:23 |
| clarkb | and just got asked to beer | 00:23 |
| clarkb | so I can review it after a beer :) | 00:23 |
| *** _david_ has quit IRC | 00:24 | |
| flashgordon | filename:logs*screen-n-cpu.txt is the right incantation right | 00:25 |
| *** melwitt has quit IRC | 00:26 | |
| flashgordon | yup | 00:26 |
| *** gokrokve_ has quit IRC | 00:27 | |
| *** CaptTofu has quit IRC | 00:28 | |
| *** CaptTofu has joined #openstack-infra | 00:28 | |
| *** masayukig has quit IRC | 00:31 | |
| *** masayukig has joined #openstack-infra | 00:32 | |
| *** CaptTofu has quit IRC | 00:32 | |
| *** wenlock has quit IRC | 00:33 | |
| clarkb | yes | 00:33 |
| *** CaptTofu has joined #openstack-infra | 00:34 | |
| *** atiwari has quit IRC | 00:55 | |
| *** mriedem has joined #openstack-infra | 00:58 | |
| *** ociuhandu has joined #openstack-infra | 01:00 | |
| *** mrodden has joined #openstack-infra | 01:00 | |
| *** CaptTofu has quit IRC | 01:01 | |
| *** CaptTofu has joined #openstack-infra | 01:02 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Update fingerprints to support grenade https://review.openstack.org/70487 | 01:02 |
| flashgordon | clarkb: ^ blamo | 01:02 |
| *** hogepodge has quit IRC | 01:05 | |
| *** CaptTofu has quit IRC | 01:06 | |
| *** yamahata has joined #openstack-infra | 01:08 | |
| *** lttrl has joined #openstack-infra | 01:09 | |
| *** emagana has quit IRC | 01:10 | |
| *** CaptTofu has joined #openstack-infra | 01:10 | |
| *** gyee has quit IRC | 01:12 | |
| *** amcrn has quit IRC | 01:20 | |
| *** masayukig has quit IRC | 01:20 | |
| *** ociuhandu has quit IRC | 01:20 | |
| openstackgerrit | Sean M. Collins proposed a change to openstack-infra/elastic-recheck: Add query for bug 1270212 https://review.openstack.org/70386 | 01:21 |
| *** hemna_ is now known as hemnabeer | 01:21 | |
| *** masayukig has joined #openstack-infra | 01:21 | |
| *** VijayT has quit IRC | 01:27 | |
| lyxus | I asked to -dev but again that might be more infra.. Does anyone know why we are using pip 1.4.1 with devstack ? | 01:29 |
| *** gokrokve has joined #openstack-infra | 01:35 | |
| *** markwash has quit IRC | 01:35 | |
| *** starmer has joined #openstack-infra | 01:42 | |
| *** CaptTofu has quit IRC | 01:43 | |
| *** CaptTofu has joined #openstack-infra | 01:44 | |
| zaro | clarkb, fungi : i just gave gave CI tools group stream events permissions globally. can you check to see if zuul has access now? | 01:45 |
| *** krtaylor has quit IRC | 01:54 | |
| *** nati_ueno has quit IRC | 01:56 | |
| *** CaptTofu has quit IRC | 01:57 | |
| *** CaptTofu has joined #openstack-infra | 01:58 | |
| fungi | lyxus: because starting with pip 1.5 any of our requirements which aren't hosted on pypi itself (of which there are several) need special options passed to pip or else it will refuse to install them for security reasons. we're still in the process of getting patches applied to the tox.ini files of affected projects to be able to work around that | 01:59 |
| fungi | lyxus: also we still have one oslo library which has not been released to pypi yet, so some projects are relying on a particular url option to be able to retrieve it from our tarballs site, and that also no longer works in pip 1.5 without special attention | 02:01 |
| fungi | zaro: i'm not sure how clarkb was checking that... i can restart the zuul service on zuul dev and see whether it starts picking up anything | 02:02 |
| *** CaptTofu has quit IRC | 02:03 | |
| *** starmer has quit IRC | 02:03 | |
| fungi | netstat on zuul-dev does at least show an established outbound socket to 29418/tcp on review-dev | 02:04 |
| fungi | i'm tailing the debug.log to see if it mentions anything | 02:05 |
| fungi | left a comment on a review on review-dev but no event was mentioned | 02:07 |
| *** starmer has joined #openstack-infra | 02:08 | |
| *** CaptTofu has joined #openstack-infra | 02:09 | |
| *** CaptTofu_ has joined #openstack-infra | 02:14 | |
| *** CaptTofu has quit IRC | 02:16 | |
| *** zehicle has joined #openstack-infra | 02:19 | |
| *** zehicle_at_dell has quit IRC | 02:22 | |
| *** masayuki_ has joined #openstack-infra | 02:25 | |
| *** masayukig has quit IRC | 02:29 | |
| *** masayukig has joined #openstack-infra | 02:37 | |
| *** VijayT has joined #openstack-infra | 02:38 | |
| *** flashgordon is now known as jog0 | 02:39 | |
| *** masayuki_ has quit IRC | 02:40 | |
| *** masayuki_ has joined #openstack-infra | 02:43 | |
| *** zehicle has quit IRC | 02:45 | |
| *** masayukig has quit IRC | 02:46 | |
| *** zehicle has joined #openstack-infra | 02:47 | |
| *** sarob has quit IRC | 02:48 | |
| *** sarob has joined #openstack-infra | 02:49 | |
| *** mgagne has quit IRC | 02:50 | |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Resolve query for bug 1258848 https://review.openstack.org/70504 | 02:51 |
| *** sarob has quit IRC | 02:53 | |
| *** masayuki_ has quit IRC | 02:58 | |
| *** markmcclain has quit IRC | 02:58 | |
| *** jaypipes has quit IRC | 02:58 | |
| *** masayukig has joined #openstack-infra | 03:00 | |
| *** praneshp has quit IRC | 03:00 | |
| *** jp_at_hp has quit IRC | 03:00 | |
| *** dcramer_ has joined #openstack-infra | 03:05 | |
| *** lttrl has quit IRC | 03:07 | |
| *** starmer has quit IRC | 03:07 | |
| *** lttrl has joined #openstack-infra | 03:07 | |
| *** starmer has joined #openstack-infra | 03:09 | |
| zaro | fungi: stream events command from zuul-dev to review-dev -> https://gerrit-documentation.storage.googleapis.com/Documentation/2.8.1/cmd-stream-events.html | 03:10 |
| fungi | zaro: right, i could emulate it but that doesn't necessarily tell me whether the zuul daemon on zuul-dev is successfully doing it | 03:13 |
| *** masayukig has quit IRC | 03:13 | |
| fungi | but it does seem to be working ow | 03:13 |
| fungi | now | 03:13 |
| fungi | 2014-02-01 02:06:47,922 DEBUG zuul.Gerrit: Change <Change 0x16ad5d0 5262,2> stat | 03:13 |
| fungi | us: NEW | 03:13 |
| fungi | entries like that in the debug log | 03:13 |
| *** masayukig has joined #openstack-infra | 03:14 | |
| *** VijayT has quit IRC | 03:16 | |
| *** CaptTofu_ has quit IRC | 03:20 | |
| *** masayukig has quit IRC | 03:24 | |
| *** talluri has joined #openstack-infra | 03:25 | |
| *** coolsvap has quit IRC | 03:26 | |
| *** CaptTofu has joined #openstack-infra | 03:31 | |
| *** markmcclain has joined #openstack-infra | 03:32 | |
| *** lttrl has quit IRC | 03:35 | |
| *** CaptTofu has quit IRC | 03:35 | |
| *** lttrl has joined #openstack-infra | 03:38 | |
| *** ryanpetrello has quit IRC | 03:40 | |
| *** talluri has quit IRC | 03:44 | |
| *** mriedem has quit IRC | 03:46 | |
| *** pcrews has quit IRC | 03:48 | |
| *** sarob has joined #openstack-infra | 03:49 | |
| *** starmer has quit IRC | 03:53 | |
| *** sarob has quit IRC | 03:54 | |
| clarkb | where is jnoller? he is posting to reddit but not on irc | 03:54 |
| *** julim has quit IRC | 03:54 | |
| clarkb | I call shenanigans | 03:54 |
| *** sarob has joined #openstack-infra | 03:59 | |
| openstackgerrit | Joshua Harlow proposed a change to openstack-infra/config: Allow tox builder to publish testrepository results https://review.openstack.org/70511 | 04:01 |
| harlowja | clarkb fungi let me know if that doesn't make sense :) | 04:01 |
| *** yamahata has quit IRC | 04:02 | |
| clarkb | harlowja: I think we should make the run-testr.sh script deal with it | 04:02 |
| clarkb | since it already deals with it | 04:02 |
| harlowja | do u mean the run-unittests.sh one? | 04:02 |
| clarkb | yeah that must be it | 04:03 |
| harlowja | k | 04:03 |
| harlowja | i was gonna do that, let me try | 04:03 |
| harlowja | seems to be sorta the same :-P | 04:03 |
| clarkb | I think that means we need to make it a little more flexible about the tox env | 04:03 |
| harlowja | right | 04:03 |
| clarkb | but it already ahs 99% of the code to do this and we don't need to duplicate | 04:03 |
| *** sarob has quit IRC | 04:03 | |
| harlowja | right, k | 04:03 |
| harlowja | fixing up | 04:03 |
| *** gokrokve has quit IRC | 04:03 | |
| *** lttrl has quit IRC | 04:07 | |
| *** lttrl has joined #openstack-infra | 04:07 | |
| *** lttrl has quit IRC | 04:11 | |
| *** lttrl has joined #openstack-infra | 04:13 | |
| *** AaronGr is now known as AaronGr_Zzz | 04:13 | |
| openstackgerrit | Joshua Harlow proposed a change to openstack-infra/config: Replace run-tox with run-unittests https://review.openstack.org/70511 | 04:18 |
| *** lttrl has quit IRC | 04:20 | |
| harlowja | k, hopefully ok with u guys | 04:21 |
| *** lttrl has joined #openstack-infra | 04:22 | |
| *** lttrl has quit IRC | 04:29 | |
| *** lttrl has joined #openstack-infra | 04:32 | |
| *** mfer has joined #openstack-infra | 04:44 | |
| *** nati_ueno has joined #openstack-infra | 04:44 | |
| *** mfer has quit IRC | 04:48 | |
| *** sarob has joined #openstack-infra | 04:50 | |
| *** sarob has quit IRC | 04:57 | |
| *** Ryan_Lane has joined #openstack-infra | 05:05 | |
| *** gokrokve has joined #openstack-infra | 05:07 | |
| *** rwsu has quit IRC | 05:08 | |
| *** david-lyle has joined #openstack-infra | 05:09 | |
| *** lcheng has joined #openstack-infra | 05:10 | |
| *** yamahata has joined #openstack-infra | 05:22 | |
| *** yamahata has quit IRC | 05:22 | |
| *** lcheng has quit IRC | 05:24 | |
| *** lcheng has joined #openstack-infra | 05:24 | |
| *** weshay has quit IRC | 05:35 | |
| *** lttrl has quit IRC | 05:35 | |
| *** gokrokve has quit IRC | 05:41 | |
| *** markmcclain has quit IRC | 05:46 | |
| *** lcheng has quit IRC | 05:49 | |
| *** sarob has joined #openstack-infra | 05:53 | |
| *** sarob has quit IRC | 05:58 | |
| *** kraman has joined #openstack-infra | 06:04 | |
| *** gokrokve has joined #openstack-infra | 06:07 | |
| *** gokrokve_ has joined #openstack-infra | 06:09 | |
| *** gokrokve has quit IRC | 06:12 | |
| *** e0ne has joined #openstack-infra | 06:19 | |
| *** lcheng has joined #openstack-infra | 06:27 | |
| *** vkozhukalov has joined #openstack-infra | 06:29 | |
| zaro | fungi: thanks. i think that's all i need to know. | 06:37 |
| clarkb | zaro: fungi: if https://issues.jenkins-ci.org/browse/JENKINS-19453 is the bug affecting our slaves maybe we should ping upstream and tell them it isn't fixed yet? | 06:40 |
| *** VijayT has joined #openstack-infra | 06:41 | |
| clarkb | I think if we see that again on 1.543 we should update the bug | 06:42 |
| zaro | ahh crap, zuul-dev still not work! | 06:44 |
| clarkb | is it at least triggering the job? | 06:44 |
| *** nati_ueno has quit IRC | 06:45 | |
| zaro | i don't have access to zuul-dev so can't tell. but i know that the jenkins job doesn't get run. | 06:45 |
| *** DinaBelova_ is now known as DinaBelova | 06:45 | |
| clarkb | well you can tell from the jenkins-dev side | 06:45 |
| clarkb | so if the jenkins-job doesn't run it isn't triggered | 06:45 |
| clarkb | zaro: is the gearman plugin configured properly? | 06:45 |
| *** banix has joined #openstack-infra | 06:45 | |
| zaro | ahh, forgot about that. let me check. | 06:46 |
| *** DinaBelova is now known as DinaBelova_ | 06:48 | |
| clarkb | zaro: any luck? if you really want to debug it tonight I can turn on the laptop and look at zuul-dev logs | 06:52 |
| clarkb | but I will let that wait for later if you are about done for the night | 06:53 |
| zaro | clarkb: so jenkins-dev log says functions were registered. however i don't have ssh access to zuul-dev so i can't actually see the functions. | 06:53 |
| clarkb | yeah I can check that nowish if you are still working on it | 06:54 |
| zaro | i'm around for a few more mins. | 06:55 |
| *** vkozhukalov has quit IRC | 06:56 | |
| *** sarob has joined #openstack-infra | 06:56 | |
| clarkb | gate-noop is registered according to geard via telnet status | 06:57 |
| clarkb | looking at zuul-dev debug log now, there are a bunch of tracebacks | 06:57 |
| clarkb | zaro: it doesn't like the layout.yaml for some reason | 06:58 |
| *** ArxCruz has quit IRC | 06:59 | |
| clarkb | but the error isn't very verbose beyond that voluptuous did a check on it and it failed | 07:00 |
| *** sarob has quit IRC | 07:01 | |
| clarkb | I am stopping zuul on zuul-dev | 07:02 |
| zaro | does layout require match job and project section? i mean do we need to add gtest-org/gtest to jobs section? | 07:03 |
| zaro | match/matching | 07:03 |
| clarkb | no | 07:04 |
| clarkb | I think the problem is there is an old version of zuul on zuul-dev that doesn't know about the triggers section in tehconfig | 07:04 |
| clarkb | going to completely blow away zuul in /usr/local/lib and reinstall | 07:04 |
| clarkb | well that didn't quite work. hmm pip install . didn't install anythin | 07:06 |
| clarkb | which may explain why it was an old version of zuul | 07:06 |
| *** dstanek has quit IRC | 07:07 | |
| zaro | that makes sense. | 07:07 |
| *** VijayT has quit IRC | 07:07 | |
| *** markwash has joined #openstack-infra | 07:07 | |
| clarkb | but pip install . does nothing and python setup.py install installs an UNKNOWNwhatever | 07:08 |
| clarkb | I think I need to upgrade pbr | 07:08 |
| *** lcheng has quit IRC | 07:09 | |
| *** vkozhukalov has joined #openstack-infra | 07:11 | |
| clarkb | well no more tracebacks but still not running the job | 07:11 |
| clarkb | zaro: try pushinga new patchset | 07:12 |
| zaro | ok, doing now.. | 07:12 |
| zaro | done | 07:13 |
| clarkb | so that worked | 07:13 |
| zaro | yeah! | 07:13 |
| clarkb | I think the comment-added filter may not work as expected | 07:13 |
| clarkb | but the patchset created did | 07:13 |
| clarkb | I am going to try a recheck no bug one more time | 07:13 |
| clarkb | ya I think we need to look at the event data for comment-added and make sure that zuul is dealing with it properly | 07:14 |
| clarkb | but the plumbing is in place now | 07:14 |
| zaro | cool! lets deal with those another day. thanks. | 07:14 |
| clarkb | yup good night | 07:15 |
| zaro | night | 07:15 |
| clarkb | and it was an old version of zuul on zuul-dev due to an old version of pbr breaking installs | 07:15 |
| clarkb | I upgraded pbr then zuul installed fine | 07:15 |
| *** VijayT has joined #openstack-infra | 07:18 | |
| clarkb | ahahaahahahahaha | 07:19 |
| clarkb | gerrit 2.4.4 - "comment":"recheck no bug" ; gerrit 2.8 - "comment":"Patch Set 4:\n\nrecheck no bug" | 07:19 |
| clarkb | so the regexes need tweaking | 07:20 |
| clarkb | but that should be easy to sort and with that I am AFK | 07:20 |
| *** e0ne has quit IRC | 07:24 | |
| *** e0ne has joined #openstack-infra | 07:25 | |
| *** kraman has quit IRC | 07:30 | |
| *** harlowja is now known as harlowja_away | 07:32 | |
| *** banix has quit IRC | 07:59 | |
| *** sarob has joined #openstack-infra | 07:59 | |
| *** markwash has quit IRC | 08:01 | |
| *** sarob has quit IRC | 08:04 | |
| *** obondarev has quit IRC | 08:05 | |
| *** plomakin has quit IRC | 08:05 | |
| *** metabro has quit IRC | 08:05 | |
| *** agordeev has quit IRC | 08:06 | |
| *** skraynev has quit IRC | 08:06 | |
| *** nadya_ has quit IRC | 08:06 | |
| *** ilyashakhat has quit IRC | 08:06 | |
| *** yolanda has joined #openstack-infra | 08:08 | |
| *** sarob has joined #openstack-infra | 08:15 | |
| *** sarob_ has joined #openstack-infra | 08:17 | |
| *** sarob has quit IRC | 08:19 | |
| *** VijayT has quit IRC | 08:20 | |
| *** sarob_ has quit IRC | 08:21 | |
| *** nprivalova has joined #openstack-infra | 08:39 | |
| *** skraynev has joined #openstack-infra | 08:39 | |
| *** saju_m has joined #openstack-infra | 08:45 | |
| *** saju_m has quit IRC | 08:48 | |
| *** saju_m has joined #openstack-infra | 08:49 | |
| *** starmer has joined #openstack-infra | 08:50 | |
| *** HenryG has quit IRC | 08:56 | |
| *** HenryG_ has joined #openstack-infra | 08:56 | |
| *** luisg has quit IRC | 09:02 | |
| *** luisg has joined #openstack-infra | 09:02 | |
| *** fbo_away is now known as fbo | 09:04 | |
| *** luis_ has joined #openstack-infra | 09:04 | |
| *** luisg has quit IRC | 09:07 | |
| *** luis_ has quit IRC | 09:09 | |
| *** luisg has joined #openstack-infra | 09:09 | |
| *** sarob has joined #openstack-infra | 09:15 | |
| *** homeless_ has joined #openstack-infra | 09:17 | |
| *** sarob has quit IRC | 09:20 | |
| *** plomakin has joined #openstack-infra | 09:23 | |
| *** metabro has joined #openstack-infra | 09:23 | |
| *** AlexF has joined #openstack-infra | 09:24 | |
| *** e0ne has quit IRC | 09:25 | |
| *** saper has quit IRC | 09:27 | |
| *** homeless has quit IRC | 09:27 | |
| *** flaper87|afk is now known as flaper87 | 09:28 | |
| *** dangers_` has joined #openstack-infra | 09:29 | |
| *** changbl has quit IRC | 09:30 | |
| *** elasticio has quit IRC | 09:30 | |
| *** mika has quit IRC | 09:30 | |
| *** malini_afk has quit IRC | 09:30 | |
| *** sdake has quit IRC | 09:30 | |
| *** isviridov has quit IRC | 09:30 | |
| *** jeremyb has quit IRC | 09:30 | |
| *** dangers_away has quit IRC | 09:30 | |
| *** shardy has quit IRC | 09:30 | |
| *** changbl has joined #openstack-infra | 09:34 | |
| *** elasticio has joined #openstack-infra | 09:34 | |
| *** mika has joined #openstack-infra | 09:34 | |
| *** malini_afk has joined #openstack-infra | 09:34 | |
| *** sdake has joined #openstack-infra | 09:34 | |
| *** jeremyb has joined #openstack-infra | 09:34 | |
| *** shardy has joined #openstack-infra | 09:34 | |
| *** isviridov has joined #openstack-infra | 09:41 | |
| *** malini_afk has quit IRC | 09:41 | |
| *** malini_afk has joined #openstack-infra | 09:41 | |
| *** malini_afk is now known as malini | 09:42 | |
| *** AlexF has quit IRC | 09:45 | |
| *** SergeyLukjanov_ is now known as SergeyLukjanov | 09:54 | |
| *** Ryan_Lane has quit IRC | 10:00 | |
| *** starmer has quit IRC | 10:05 | |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 10:07 | |
| *** SergeyLukjanov is now known as SergeyLukjanov_ | 10:15 | |
| *** sarob has joined #openstack-infra | 10:15 | |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 10:17 | |
| *** sarob has quit IRC | 10:20 | |
| *** morganfainberg is now known as morganfainberg|z | 10:22 | |
| *** SergeyLukjanov_ is now known as SergeyLukjanov | 10:37 | |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 10:38 | |
| openstackgerrit | Ruslan Kamaldinov proposed a change to openstack-infra/storyboard: Get rid of Django https://review.openstack.org/66016 | 10:49 |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 10:57 | |
| *** SergeyLukjanov is now known as SergeyLukjanov_ | 11:00 | |
| *** sarob has joined #openstack-infra | 11:15 | |
| *** sarob_ has joined #openstack-infra | 11:17 | |
| openstackgerrit | Yuriy Taraday proposed a change to openstack-infra/git-review: Attach known_hosts to failed tests' details https://review.openstack.org/70527 | 11:19 |
| *** sarob has quit IRC | 11:20 | |
| *** sarob_ has quit IRC | 11:22 | |
| *** e0ne has joined #openstack-infra | 11:30 | |
| *** lttrl has joined #openstack-infra | 11:39 | |
| openstackgerrit | Yuriy Taraday proposed a change to openstack-infra/git-review: Attach known_hosts to failed tests' details https://review.openstack.org/70527 | 11:40 |
| openstackgerrit | Yuriy Taraday proposed a change to openstack-infra/git-review: Attach known_hosts to failed tests' details https://review.openstack.org/70527 | 11:43 |
| *** ihrachys|fosdem has joined #openstack-infra | 11:55 | |
| *** lttrl has quit IRC | 11:56 | |
| ihrachys|fosdem | hi all. I have two issues with my gerrit account, and I hope infra guys can help me with this. First, there are two duplicate entries for Ihar Hrachyshka <ihrachys@*.com> in reviewers list; second, it fails when a user tries to add me to the list (it says: Ihar Hrachyshka <ihrachys@***.com> is neither a registered user nor a group.) | 11:58 |
| *** malini is now known as malini_afk | 12:07 | |
| *** sarob has joined #openstack-infra | 12:15 | |
| *** sarob has quit IRC | 12:20 | |
| *** e0ne has quit IRC | 12:43 | |
| *** vogxn has joined #openstack-infra | 12:55 | |
| *** yolanda has quit IRC | 13:00 | |
| *** saju_m has quit IRC | 13:06 | |
| *** sarob has joined #openstack-infra | 13:15 | |
| *** yamahata has joined #openstack-infra | 13:18 | |
| *** sarob has quit IRC | 13:20 | |
| openstackgerrit | Alexandre Levine proposed a change to openstack-infra/config: Adding jenkins jobs https://review.openstack.org/70530 | 13:33 |
| *** salv-orlando has quit IRC | 13:42 | |
| *** mriedem has joined #openstack-infra | 13:43 | |
| *** cody-somerville_ has joined #openstack-infra | 13:54 | |
| *** cody-somerville has quit IRC | 13:56 | |
| *** DinaBelova_ is now known as DinaBelova | 14:01 | |
| *** vogxn has quit IRC | 14:02 | |
| *** DinaBelova is now known as DinaBelova_ | 14:13 | |
| *** sarob has joined #openstack-infra | 14:15 | |
| *** sarob has quit IRC | 14:20 | |
| *** CaptTofu has joined #openstack-infra | 14:24 | |
| *** e0ne has joined #openstack-infra | 14:34 | |
| *** saju_m has joined #openstack-infra | 14:41 | |
| *** dstanek has joined #openstack-infra | 15:01 | |
| *** ryanpetrello has joined #openstack-infra | 15:05 | |
| *** yassine has joined #openstack-infra | 15:10 | |
| *** sarob has joined #openstack-infra | 15:15 | |
| *** sarob has quit IRC | 15:21 | |
| *** ryanpetrello has quit IRC | 15:29 | |
| *** salv-orlando has joined #openstack-infra | 15:31 | |
| anteaya | ihrachys|fosdem: ensure you are signed into your gerrit account, go to the settings page, what is the account id you see on that page? | 15:33 |
| anteaya | 1 in the gate, 3 in check, I can't remember how long it has been since I have seen that | 15:37 |
| *** salv-orlando has quit IRC | 15:39 | |
| *** salv-orlando has joined #openstack-infra | 15:49 | |
| *** salv-orlando has quit IRC | 15:51 | |
| *** e0ne has quit IRC | 15:55 | |
| openstackgerrit | Guido Günther proposed a change to openstack-infra/jenkins-job-builder: Support site monitor publisher https://review.openstack.org/69290 | 16:03 |
| *** markwash has joined #openstack-infra | 16:04 | |
| ihrachys|fosdem | anteaya: 9656 | 16:09 |
| anteaya | thanks | 16:09 |
| anteaya | once someone with access to the gerrit db appears they can see what is happening with the account at that id and let you know | 16:10 |
| anteaya | usually it is fungi but he is deserving of some time off line so please be patient | 16:11 |
| ihrachys|fosdem | anteaya: ok, thank you | 16:11 |
| anteaya | jeblair would have access and is at fosdem so if you see him, you might ask him about it | 16:11 |
| anteaya | this sort of thing crops up from time to time, but we do end up getting it straightened out | 16:12 |
| *** sarob has joined #openstack-infra | 16:15 | |
| *** banix has joined #openstack-infra | 16:17 | |
| *** gokrokve has joined #openstack-infra | 16:18 | |
| *** sarob has quit IRC | 16:20 | |
| *** gokrokve_ has quit IRC | 16:20 | |
| *** e0ne has joined #openstack-infra | 16:27 | |
| *** CaptTofu has quit IRC | 16:30 | |
| *** talluri has joined #openstack-infra | 16:33 | |
| *** gokrokve has quit IRC | 16:36 | |
| *** gokrokve_ has joined #openstack-infra | 16:38 | |
| *** fbo is now known as fbo_away | 16:45 | |
| *** banix has quit IRC | 16:45 | |
| *** blitzrage is now known as leifmadsen | 16:47 | |
| *** AlexF has joined #openstack-infra | 16:48 | |
| *** e0ne has quit IRC | 16:51 | |
| *** praneshp has joined #openstack-infra | 16:58 | |
| *** david-lyle has quit IRC | 16:59 | |
| *** saju_m has quit IRC | 17:00 | |
| *** praneshp_ has joined #openstack-infra | 17:00 | |
| *** saju_m has joined #openstack-infra | 17:01 | |
| *** praneshp has quit IRC | 17:03 | |
| *** praneshp_ is now known as praneshp | 17:03 | |
| *** dstufft has quit IRC | 17:07 | |
| *** gokrokve_ has quit IRC | 17:09 | |
| *** gokrokve has joined #openstack-infra | 17:10 | |
| *** gokrokve has quit IRC | 17:14 | |
| *** sarob has joined #openstack-infra | 17:15 | |
| *** markwash has quit IRC | 17:15 | |
| *** alexpilotti has joined #openstack-infra | 17:16 | |
| fungi | ihrachys|fosdem: i'm taking a look now | 17:16 |
| * fungi slept in, it being saturday and all | 17:16 | |
| *** sarob has quit IRC | 17:20 | |
| *** AlexF has quit IRC | 17:24 | |
| *** rustlebee is now known as russellb | 17:26 | |
| fungi | ihrachys|fosdem: i see two accounts for you, ids are 9656 and 9800. both have your redhat e-mail address associated with them. 9656 was created and signed the icla on december 10, has an ssh username of ihrachys and two ssh keys. 9800 was created on december 23, did not sign the icla, has no ssh username but has your gmail address as an alternate e-mail address | 17:27 |
| *** yassine has quit IRC | 17:28 | |
| *** yassine has joined #openstack-infra | 17:28 | |
| *** yassine has quit IRC | 17:32 | |
| fungi | ihrachys|fosdem: since it looks like you've mainly been using the first account, i've reassigned the openid url for 9800 to 9656 instead and blanked out the e-mail address for 9800 so that it doesn't cause reviewer selection problems in gerrit later | 17:32 |
| *** yassine has joined #openstack-infra | 17:32 | |
| *** yassine has quit IRC | 17:32 | |
| fungi | ihrachys|fosdem: you may want to log out of gerrit's web interface and log back in again though, if you're currently logged in | 17:33 |
| fungi | i'm starting to think at least one of the ways this issue gets triggered is when users switch between the e-mail address they use to authenticate to launchpad, which would explain why sometimes we see accounts alternate back and forth | 17:34 |
| fungi | because it looks like launchpad ends up associating different e-mail addresses to different openid urls even when they're for the same lp account | 17:35 |
| *** AlexF has joined #openstack-infra | 17:42 | |
| *** zul has quit IRC | 17:44 | |
| *** emagana has joined #openstack-infra | 17:45 | |
| *** starmer has joined #openstack-infra | 17:48 | |
| *** VijayT has joined #openstack-infra | 17:48 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Convert failed_job into an object https://review.openstack.org/70537 | 17:51 |
| *** AlexF has quit IRC | 17:56 | |
| devananda | fungi: hi! have ya'll been looking into a gate issue with nova-compute failures from libguestfs? | 17:57 |
| *** sarob has joined #openstack-infra | 17:57 | |
| devananda | logstash says it started ~18 hours ago | 17:58 |
| devananda | http://bit.ly/1eGK8Xu | 17:59 |
| fungi | devananda: i haven't, but maybe the fine folks in #openstack-qa have been tracking it | 17:59 |
| devananda | ack | 17:59 |
| fungi | devananda: i think nova maybe landed a change yesterday to use that as a means of avoiding the nbd kernel issues we were seeing in neutron jobs? | 18:00 |
| devananda | fungi: i see - thanks! | 18:00 |
| *** emagana has quit IRC | 18:04 | |
| openstackgerrit | A change was merged to openstack/requirements: Have tox install via setup.py develop https://review.openstack.org/66549 | 18:13 |
| *** cody-somerville_ has quit IRC | 18:13 | |
| * fungi is afk again for a while | 18:16 | |
| devananda | fungi: when you're back, can you point me to info on how to log a bug with elasticrecheck? | 18:27 |
| *** talluri has quit IRC | 18:29 | |
| *** talluri has joined #openstack-infra | 18:30 | |
| *** Ryan_Lane has joined #openstack-infra | 18:31 | |
| *** talluri has quit IRC | 18:34 | |
| clarkb | devananda: clone elasticrecheck, and add a file in the queries dir with the bugnumber.yaml | 18:37 |
| *** yolanda has joined #openstack-infra | 18:50 | |
| *** starmer has quit IRC | 18:54 | |
| *** starmer has joined #openstack-infra | 18:54 | |
| *** AaronGr_Zzz is now known as AaronGr | 18:59 | |
| *** AaronGr is now known as AaronGr_Zzz | 19:00 | |
| openstackgerrit | Devananda van der Veen proposed a change to openstack-infra/config: Make Ironic's devstack-gate tests more like Nova's https://review.openstack.org/70544 | 19:06 |
| devananda | clarkb: think that ^ is going to help our gate? or are there other neutron'ish bits in ironic's devstack-gate config that i need to hunt down? | 19:07 |
| clarkb | devananda: does ironic support config drive? | 19:08 |
| devananda | clarkb: no | 19:08 |
| devananda | clarkb: we rely on cloud-init | 19:08 |
| *** saju_m has quit IRC | 19:08 | |
| clarkb | devananda: and that will run a lot of extra tests in the gate. probably want to try it as an experimental job first just to have a sanity check | 19:08 |
| devananda | "extra tests" ...? | 19:09 |
| clarkb | devananda: yes an order of magnitude more tests run iirc | 19:09 |
| clarkb | its a couple hundred vs 1200 or some such | 19:09 |
| devananda | ah | 19:09 |
| clarkb | you can see the output of tempest in the other jobs that run full for numbers | 19:09 |
| clarkb | devananda: so you don't do file injections or config drive and rely on cloud init? In that case I think you want to disable both config drive and file injection | 19:10 |
| clarkb | devananda: which will require updating russellb's devstack/lib/nova patch | 19:10 |
| clarkb | (assuming you dno't instead swap out libvirt first) | 19:10 |
| devananda | so i should just remove the SERIAL flag, but not enable FULL, it sounds like | 19:11 |
| clarkb | well you need one or the other | 19:11 |
| devananda | heh | 19:11 |
| clarkb | there may be a non serial smoke, I don't remember | 19:11 |
| *** emagana has joined #openstack-infra | 19:11 | |
| devananda | re: libvirt, mirantis folks are working on adding hooks to devstack for the nova-ironic driver | 19:12 |
| devananda | there's a bit of chicken and egg. we need to land that in devstack before we can test the nova-ironic driver. but we can't enable them in our own gate until taht driver lands in nova | 19:12 |
| devananda | i still get confused by how we're supposed to test cross-project deps like this | 19:13 |
| *** markwash has joined #openstack-infra | 19:13 | |
| *** vkozhukalov has quit IRC | 19:16 | |
| clarkb | you put them in devstack, then you flip the bit in devstack gate. Since devstack gate is self testing that will gate itself on the ability to test the nova-ironic driver | 19:17 |
| clarkb | you should able to use that method to test that the full stack works | 19:18 |
| openstackgerrit | Devananda van der Veen proposed a change to openstack-infra/config: Make Ironic tests stop using file injection https://review.openstack.org/70544 | 19:20 |
| devananda | clarkb: better? | 19:21 |
| clarkb | devananda: ya that should do it | 19:23 |
| devananda | great | 19:23 |
| *** gokrokve has joined #openstack-infra | 19:30 | |
| *** yolanda has quit IRC | 19:38 | |
| *** locke1051 is now known as locke105 | 19:44 | |
| devananda | clarkb: are any projects gating on neutron today, besides neutron itself? | 19:47 |
| clarkb | yes all of them | 19:48 |
| devananda | eh? | 19:49 |
| *** mriedem has quit IRC | 19:50 | |
| devananda | ah, right. that was the cause of all the gate back-log last week | 19:51 |
| devananda | somehow i thought many of them were non-voting | 19:53 |
| *** e0ne has joined #openstack-infra | 19:55 | |
| *** CaptTofu has joined #openstack-infra | 19:56 | |
| *** emagana has quit IRC | 19:57 | |
| clarkb | devananda: so the failure rate on this injection thing doesn't appear to be 100% or is it? | 19:59 |
| clarkb | there are neutron jobs passing in the gate | 19:59 |
| *** mriedem has joined #openstack-infra | 20:04 | |
| devananda | clarkb: logstash tells me that check-tempest-dsvm-neutron has 403 fails and 24 successes in the last 4 hours | 20:06 |
| clarkb | those arent job fails though | 20:07 |
| clarkb | oh you mean in general | 20:07 |
| devananda | of those 403, only 6 also match "libguestfs" | 20:07 |
| *** markwash has quit IRC | 20:08 | |
| anteaya | I'm late to the discussion about guestfs but salv-orlando filed this bug about it: https://bugs.launchpad.net/nova/+bug/1275267 | 20:09 |
| anteaya | devananda: did you also file a bug? | 20:09 |
| devananda | anteaya: https://bugs.launchpad.net/nova/+bug/1275301 | 20:10 |
| clarkb | so I <3 that everyone is doing the logstash thing | 20:10 |
| clarkb | but can you guys be sure to restrict by filename if the log shows up in only one file? | 20:10 |
| lifeless | devananda: configdrive on will turn metadata off - that will mean you dont' get keys installed and tempest wont be able to ssh into the cirros image | 20:10 |
| clarkb | otherwise elastisearch has to load tons of data into memory | 20:10 |
| lifeless | devananda: so I think that that patch is invalid | 20:10 |
| clarkb | and these boxes are memory contrained | 20:10 |
| devananda | clarkb: ah, good to know, thanks | 20:11 |
| lifeless | clarkb: have we landed russelb's patch, or was guestfs broken too? | 20:12 |
| anteaya | clarkb: I can work towards that, is there anyway of finding out if the error is only in one log file? The only way I know of is to check each file. | 20:12 |
| clarkb | devananda: https://jenkins06.openstack.org/job/check-tempest-dsvm-full/428/consoleFull I have never been so happy to see a fail. going to let zuul report to my change then push a second patchset with what I hope will fix it | 20:12 |
| clarkb | lifeless: russellb's patch isn't being used or at least isn't used in the way everyone expected | 20:12 |
| anteaya | is there some link on the left hand side of logstash that would be helpful? | 20:13 |
| clarkb | anteaya: you can do a small search like you have been doing then check if results show up in more than one file | 20:13 |
| clarkb | anteaya: yes left hand side, click filename and there should be options to get statistics | 20:13 |
| *** dstufft has joined #openstack-infra | 20:13 | |
| devananda | clarkb: woot | 20:13 |
| clarkb | do this over 60 minutes and not all time please :) | 20:13 |
| clarkb | lifeless: basically all of the tests that are not neutron and ironic use config drive | 20:13 |
| lifeless | devananda: https://review.openstack.org/#/c/70239/ - fixing that is what I would suggest | 20:13 |
| clarkb | lifeless: nuetron and ironic use file injection. file injection is broken with guestfs | 20:14 |
| anteaya | found it, filename | 20:14 |
| lifeless | clarkb: yes, I know; neutron doesn't use config drive because metadata was being broken on a monthly basis without testing. | 20:14 |
| anteaya | obvious really | 20:14 |
| lifeless | clarkb: ironic can't hand metadata in config drive, so all the tempest tests that depend on logging into a VM... won't work. | 20:14 |
| clarkb | lifeless: rights its fine | 20:15 |
| clarkb | but it was a gap in the testing I think | 20:15 |
| devananda | clarkb: ironic actually uses metadata. it's a cart-before-horse that our tests are, today, using either configdrive or fileinjection | 20:15 |
| clarkb | I am working on a fix | 20:15 |
| anteaya | clarkb: thanks | 20:15 |
| lifeless | devananda: failing that, add a localrc with a nova config option to set [libvirt]inject_partition=-2 | 20:15 |
| clarkb | lifeless: devananda: so that would fix it too | 20:15 |
| * clarkb is trying to see if the libguestfs docs for precise actually fix the problem | 20:16 | |
| clarkb | regardless of whether or not you want to use file injection | 20:16 |
| lifeless | I suspect the root bug here is that the libvirt driver is still being used even with ironic enabled | 20:16 |
| lifeless | devananda: ^ ? | 20:16 |
| clarkb | lifeless: that is part of it, but only for ironic, neutron is still broken | 20:16 |
| lifeless | clarkb: I know | 20:17 |
| devananda | lifeless: so, that's also true. but we don't have a nova-ironic driver in trunk today | 20:17 |
| devananda | lifeless: we should really be not testing Nova in our gate at the moment | 20:17 |
| clarkb | https://review.openstack.org/#/c/70546/ if that pans out we may have a fix for neutron | 20:17 |
| devananda | lifeless: re your comment on 70544, i'm trying to -enable configdriver. it's currently disabled | 20:18 |
| *** sarob has quit IRC | 20:21 | |
| anteaya | clarkb: so you point of 70546 is to see if you can get it passing consistently in check? | 20:22 |
| *** sarob has joined #openstack-infra | 20:22 | |
| lifeless | devananda: I think I got the sense inverted or something | 20:22 |
| clarkb | anteaya: yes, then if it does we can work on cleaning that up in devstack because right now it is a bit of a hack | 20:23 |
| lifeless | devananda: my memory of the config drive stuff - may be wrong - is that config drive on implies metadata off, for devstack. | 20:23 |
| devananda | lifeless: ugh... | 20:24 |
| lifeless | echo "FORCE_CONFIG_DRIVE=always" >>localrc -> iniset $NOVA_CONF DEFAULT force_config_drive "$FORCE_CONFIG_DRIVE" | 20:24 |
| clarkb | right so that is not enabled on ironic or neutron tests | 20:25 |
| *** CaptTofu has quit IRC | 20:25 | |
| clarkb | they both use file injection today | 20:25 |
| anteaya | clarkb: okay great | 20:25 |
| lifeless | clarkb: yes, the question is whether the metadata server is also available | 20:25 |
| *** sarob has quit IRC | 20:26 | |
| devananda | clarkb: so eventually, ironic should be tested with VIRT_DRIVER=ironic, not VIRT_DRIVER=libvirt | 20:26 |
| devananda | clarkb: even though taht's not available at the moment, adding a stub to devstack to nova would avoid this problem | 20:27 |
| devananda | since it would then not exercise the file injection path (or the config drive path) in libvirt | 20:27 |
| clarkb | devananda: but it wouldn't exercise anything other than the noop driver | 20:27 |
| devananda | this issue in ironic's gate is an artefact of the libvirt tests running when they shouldn't | 20:27 |
| clarkb | devananda: partially, file injections is broken | 20:28 |
| devananda | clarkb: right. and at the moment, that's appropriate | 20:28 |
| devananda | clarkb: but ironic doesn't support file injection. | 20:28 |
| clarkb | devananda: but neutron does | 20:28 |
| clarkb | and neutron is part of the same gate | 20:28 |
| devananda | clarkb: file injection in libvirt is broken. this /should/ show up in the nova libvirt tests | 20:28 |
| clarkb | devananda: it does | 20:28 |
| clarkb | that is what I am saying | 20:28 |
| devananda | neutron is not part of ironic's gate (today) | 20:28 |
| clarkb | it shows up in the neutron libvirt tests | 20:28 |
| devananda | right | 20:28 |
| devananda | so fixing the guestfs issue is important for neutron libvirt tests | 20:29 |
| devananda | it's actually not relevant to ironic - -we shouldn't be hitting it | 20:29 |
| devananda | i'm not saying it's not a problem | 20:29 |
| clarkb | lifeless: best I can tell metadata is a defualt service with 4 workers | 20:29 |
| lifeless | ok | 20:30 |
| devananda | there's 2 issues: a) file injection in libvirt is broken b) ironic d-g is exercising libvirt when it shouldn't be | 20:30 |
| lifeless | so it will be available for ironic | 20:30 |
| clarkb | lifeless: I think so | 20:30 |
| *** sarob has joined #openstack-infra | 20:30 | |
| lifeless | I don't think fixing guestfs is important: we're on a path to deprecate file injection | 20:30 |
| devananda | lifeless: exactly | 20:30 |
| clarkb | lifeless: it is important if we test neutron with it and need to deprecate for a cycle or two | 20:31 |
| *** e0ne has quit IRC | 20:31 | |
| devananda | which is why i'm looking for a path to (b) | 20:31 |
| anteaya | neutron needs to be able to merge code | 20:31 |
| lifeless | I want to know that neutron / metadata work without injection or config drive | 20:31 |
| clarkb | lifeless: well then add that to a test | 20:31 |
| anteaya | we have been unable to merge anything for over 2 weeks | 20:31 |
| anteaya | at this rate we won't get anything into i3 | 20:31 |
| clarkb | right now we test two things. config drive and file injection. If we need to test a third thing we change a test | 20:31 |
| lifeless | anteaya: sure, and thats why I have a patch up to disable file injection | 20:31 |
| anteaya | in neutron? | 20:32 |
| lifeless | clarkb: no, we test config drive and metadata - *noone* realised file injection was still happening | 20:32 |
| lifeless | clarkb: until this blew up | 20:32 |
| clarkb | lifeless: no we actually test config drive and file injections :) | 20:32 |
| clarkb | I get what you are saying | 20:32 |
| lifeless | clarkb: the explicit intent - and you can read the discussion we had right here a few months back was to test config drive + metadata. | 20:32 |
| clarkb | but there is the code that is tested which should work because it is gated and there is the other stuff | 20:32 |
| lifeless | clarkb: we disabled config drive *to test metadata* | 20:32 |
| clarkb | if you want to change the gate that is a different problem | 20:32 |
| *** markwash has joined #openstack-infra | 20:33 | |
| lifeless | clarkb: I feel like you're not listening | 20:33 |
| clarkb | lifeless: I get it | 20:33 |
| lifeless | clarkb: do I need to drag up the IRC discussion about this? | 20:33 |
| clarkb | no | 20:33 |
| clarkb | I totally get it | 20:33 |
| devananda | clarkb: are you saying no one tests metadata today? | 20:33 |
| lifeless | clarkb: ok; so you get that we test metadata. | 20:33 |
| clarkb | devananda: yes | 20:33 |
| devananda | ...... | 20:33 |
| lifeless | clarkb: what you are saying contradicts what I'm saying. | 20:33 |
| lifeless | clarkb: you saying you get it contradicts that. | 20:33 |
| lifeless | clarkb: I am now 100% confused. | 20:34 |
| clarkb | lifeless: my reading of the tests is that metadata is not being tested | 20:34 |
| clarkb | because file injection is enabled and config drive is disabled | 20:34 |
| clarkb | in the other tests config drive is enabled | 20:34 |
| lifeless | clarkb: are you aware that even when file injection is enabled metadata gets tested ? | 20:34 |
| *** sarob has quit IRC | 20:34 | |
| clarkb | lifeless: are there two different sets of tests? | 20:35 |
| lifeless | clarkb: there are three data sources for cloud-init | 20:35 |
| lifeless | clarkb: (for this discussion) | 20:35 |
| *** dstanek is now known as dstanek_afk | 20:35 | |
| clarkb | so we have a binary config drive or file injection in libvirt/driver.py | 20:35 |
| clarkb | and there is a third cloud-init + metadata that always happens? | 20:35 |
| lifeless | clarkb: config drive; local file; ec2 metadata. | 20:35 |
| *** mgagne has joined #openstack-infra | 20:35 | |
| lifeless | clarkb: file injection *does not* setup local file, it just mutates the OS image directly. | 20:36 |
| *** sarob has joined #openstack-infra | 20:36 | |
| clarkb | let us backup. we are testing config drive and file injection today. Are we also testing metadata along side both of those? | 20:36 |
| lifeless | clarkb: from a VM perspective then, if config drive is on, cloud-init will not talk to the ec2 metadata server. | 20:36 |
| clarkb | the toggles being toggled affect config drive and file injection not metadata as far as I can tell | 20:37 |
| lifeless | clarkb: if config drive is off (and just ignore file injection _please_) | 20:37 |
| lifeless | clarkb: then cloud-init will require the metadata server | 20:37 |
| clarkb | right so we test config drive and file injection + metadata? | 20:37 |
| clarkb | and you want it to be config drive and metadata (no file injection) | 20:38 |
| lifeless | clarkb: the *intent* of the current gate configuration is to test these two interactions: config drive + metadata, file injection being exercised was incidental. | 20:38 |
| lifeless | clarkb: and thus seans proposal to *turn it off* [by turning on config drive], which also has the negative effect of turning off tests of *metadata*, which we *do* want to test. | 20:38 |
| lifeless | clarkb: your definition of test and mine seem to be different. | 20:39 |
| clarkb | lifeless: I think my confusion lies around russellb's change explicitly enabling file injection | 20:39 |
| clarkb | his intent seemed to be to test file injection | 20:39 |
| lifeless | clarkb: which one, the one to install guestfs? | 20:39 |
| clarkb | yes | 20:39 |
| lifeless | thats an attempt to fix the neutron gate without changing anything else | 20:39 |
| clarkb | but it is breaking the neutron gate | 20:40 |
| lifeless | it was broken before, because the default in nova is to do file injection with nbd, which crashes the kernel. | 20:40 |
| *** sarob has quit IRC | 20:40 | |
| clarkb | so why not disable file injection completely? | 20:41 |
| clarkb | this is where I am confused | 20:41 |
| lifeless | we will once I or someone else figures out why my patch *to do that* fails. | 20:41 |
| lifeless | the blueprint is approved | 20:41 |
| clarkb | no why didn't russellb do that? just an oversight? | 20:42 |
| lifeless | to disable by default | 20:42 |
| clarkb | gotcha so must've been an oversight. Now I understand | 20:42 |
| lifeless | clarkb: I would speculate that shortest-path-to-fix-things was to switch a known broken thing (nbd) to a presumed working thing (guestfs) | 20:42 |
| lifeless | now, if I could figure out how to put the ini setting in directly in d-g, I'd happily put up a patch to disable file injection - but note that grenade shows this failing - and we don't know why yet | 20:43 |
| lifeless | nova-compute just gives up the ghost after a few VMs | 20:43 |
| clarkb | if guestfs is deprecated wouldn't it be better to test it until it is removed? | 20:43 |
| lifeless | https://review.openstack.org/#/c/70239/ | 20:43 |
| lifeless | clarkb: thats a different discussion :P | 20:44 |
| lifeless | clarkb: in tripleo I'm making an opinionated stance of not turning on file injection, it's not a sane default | 20:44 |
| clarkb | I can agree with that | 20:44 |
| clarkb | btu I can also see value in testing it until it is completely removed | 20:45 |
| lifeless | sure | 20:45 |
| lifeless | though thats really up to sean/russell - until I asked for metadata to be tested, config drive was the only path tested here. | 20:45 |
| lifeless | So adding in explicit testing of file injection is IMO really a different discussion | 20:46 |
| clarkb | wait no. neutron tested file injection with nbd | 20:46 |
| clarkb | (it didn't work reliably either) | 20:46 |
| lifeless | no, it was doing config drive | 20:46 |
| lifeless | about 3 months or something back, I whinged about metadata being broken for the Nth time in as many months, and we identified that nothing was testing metadata. | 20:46 |
| lifeless | the solution was to move one test configuration to have config drive off. | 20:47 |
| lifeless | We chose neutron because neutron's metadata agent wouldn't be tested otherwise. | 20:47 |
| clarkb | 082759bbb99ca5e34e1709971a692fb855467f3c that commit | 20:47 |
| lifeless | the fact that file injection ended up being used was an oversight IIRC. | 20:47 |
| clarkb | I see | 20:48 |
| lifeless | see, *no* mention of file injection | 20:48 |
| lifeless | I quote for the channel... "This scattering should be reasonably representative to feel that | 20:49 |
| lifeless | we are exercising config_drive and metadata server well." | 20:49 |
| lifeless | but we need to figure out why nova compute dies in https://review.openstack.org/#/c/70239/ | 20:49 |
| lifeless | it's a genuine failure | 20:49 |
| lifeless | or | 20:49 |
| clarkb | right, and sdague indicates it is a real problem | 20:49 |
| lifeless | perhaps we can disabled file injection in devstack but not in grenade... | 20:49 |
| clarkb | lifeless: well I wonder | 20:50 |
| clarkb | lifeless: russellb explicitly enables file injection in devstack | 20:50 |
| sdague | lifeless: that fact that that failed, is a real issue | 20:50 |
| clarkb | which completely undoes your defaults | 20:50 |
| clarkb | I would try fixing it not in nova but in devstack first (this ignores grenade) | 20:50 |
| sdague | because it means something in the havana config doesn't work with your new default | 20:50 |
| sdague | so it's actually a real upgrade issue | 20:51 |
| lifeless | sdague: totally | 20:52 |
| lifeless | sdague: I am not suggesting ignoring it | 20:52 |
| lifeless | sdague: I'm wondering if doing two things: a) disable file injection in the neutron job, then async, figure out the issue with changing the default | 20:52 |
| *** sarob has joined #openstack-infra | 20:53 | |
| sdague | so we merged the guestfs changes | 20:53 |
| lifeless | sdague: and its shown that guestfs is broken, right? | 20:53 |
| sdague | no, we merged those fixes | 20:53 |
| lifeless | oh, cool | 20:53 |
| clarkb | guestfs is broken for file injection | 20:54 |
| sdague | clarkb: .... ? | 20:54 |
| sdague | clarkb: pointer? | 20:54 |
| sdague | or did I miss a thing | 20:54 |
| clarkb | https://bugs.launchpad.net/nova/+bug/1275267 | 20:54 |
| clarkb | sdague: we missed that russellb's change to enable guestfs explicitly enables file injection if config drive is not enabled | 20:55 |
| sdague | oh, that's new | 20:55 |
| clarkb | and guestfs doesn't work on precise without extra bit fiddling according to their FAQ | 20:55 |
| lifeless | clarkb: explicitly or implicitly? | 20:55 |
| lifeless | clarkb: cause nova *defaults* to enabled | 20:55 |
| sdague | the nova defaults changed | 20:55 |
| clarkb | sdague: https://review.openstack.org/#/c/70546/ is my attempt as seeing a fix for file injection | 20:55 |
| clarkb | lifeless: sdague: if config drive is enabled file injection is not used even if enabled | 20:55 |
| clarkb | so implicit | 20:56 |
| clarkb | see https://git.openstack.org/cgit/openstack/nova/tree/nova/virt/libvirt/driver.py#n2564 | 20:56 |
| lifeless | clarkb: right, but you said explicit. This confused me :) | 20:56 |
| clarkb | lifeless: russellb's change is explicit | 20:56 |
| clarkb | sets it to -1 | 20:56 |
| clarkb | which is why your nova change won't actually tset what you want to test aiui | 20:56 |
| clarkb | because devstack will just override you | 20:57 |
| sdague | right | 20:57 |
| *** sarob has quit IRC | 20:57 | |
| lifeless | nuts ok so if the approach is a big hammer we can do that :P | 21:00 |
| lifeless | https://review.openstack.org/70549 | 21:00 |
| lifeless | we'll see if this breaks in teh same way | 21:00 |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Add query for nova/devstack bug 1275267 https://review.openstack.org/70550 | 21:01 |
| *** e0ne has joined #openstack-infra | 21:02 | |
| anteaya | devananda: do you agree https://bugs.launchpad.net/nova/+bug/1275301 is a dup of 1275267? | 21:03 |
| lifeless | if we need to support both options we'll need to make this an option in devstack then control it from gevstackgate | 21:03 |
| lifeless | devstackgate | 21:03 |
| *** e0ne_ has joined #openstack-infra | 21:03 | |
| *** e0ne has quit IRC | 21:03 | |
| anteaya | devananda: I'd like to know what you think before I vote on patch 70550 | 21:04 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for nova/devstack bug 1275267 https://review.openstack.org/70550 | 21:04 |
| anteaya | too late for me | 21:04 |
| devananda | anteaya: i'm the one who filed 1275301,a nd marked it as a dup :) | 21:05 |
| sdague | anteaya: sorry, I tend to go liberal on approving the fingerprints, especially if they come from mriedem, as he's pretty solid on them. | 21:05 |
| anteaya | devananda: ah cool, thanks | 21:06 |
| anteaya | sdague: np | 21:06 |
| sdague | if you think it's a problem please let me know | 21:06 |
| anteaya | glad to have it in | 21:06 |
| anteaya | not at all | 21:06 |
| lifeless | sdague: help me understand grenade a little | 21:06 |
| lifeless | sdague: http://logs.openstack.org/39/70239/1/check/check-grenade-dsvm/f1196f7/console.html.gz console and http://logs.openstack.org/39/70239/1/check/check-grenade-dsvm/f1196f7/logs/new/screen-n-cpu.txt.gz#_2014-01-31_03_05_17_115 nova compute | 21:06 |
| devananda | lifeless: given your discussion with clarkb , do you still feel the same (-1) about https://review.openstack.org/#/c/70544/2 ? | 21:06 |
| anteaya | at some point, I would like to be more useful for elastic-recheck so when that time comes, I'll let you know | 21:06 |
| lifeless | sdague: nova compute stops cold adt 2014-01-31 03:05:17.115 DEBUG nova.openstack.common.lockutils [req-aa21788b-51cc-4121-9318-a1d210c8be30 ImagesOneServerTestJSON-1187612811 ImagesOneServerTestJSON-134614619] Semaphore / lock released "copy_qcow2_image" inner /opt/stack/new/nova/nova/openstack/common/lockutils.py:252 | 21:07 |
| sdague | lifeless: ok, that seems bad. Deadlock? | 21:07 |
| sdague | lifeless: they way I read these logs the last time | 21:08 |
| sdague | it looks like at some point guest allocations stop completing | 21:08 |
| *** e0ne_ has quit IRC | 21:08 | |
| sdague | or maybe don't tear down | 21:08 |
| sdague | eventually the scheduler runs out of headroom, and stops scheduling | 21:09 |
| *** gokrokve has quit IRC | 21:09 | |
| sdague | but that was about 20 minutes of decyphering after your recheck, because it failed all the jobs, so it wasn't likely to be a race | 21:09 |
| *** CaptTofu has joined #openstack-infra | 21:12 | |
| lifeless | sdague: so the way I read it, nova copmute crashed | 21:13 |
| lifeless | sdague: otherwise we'd see update_available_resource still running | 21:14 |
| lifeless | but 2014-01-31 03:04:53.571 5831 DEBUG nova.openstack.common.lockutils [-] Semaphore / lock released "update_available_resource" inner /opt/stack/new/nova/nova/openstack/common/lockutils.py:252 | 21:15 |
| lifeless | is the last hit for it, which is a release | 21:15 |
| lifeless | sdague: without the compute process advertising to the scheduler, the scheduler will see 0 hosts available | 21:16 |
| lifeless | sdague: and from there on give the same top level sumptoms | 21:16 |
| sdague | so we're actually capturing screen output, if it crashed we usually get a stack trace | 21:17 |
| sdague | or did we turn that off in the gate? | 21:17 |
| clarkb | I don't think we did | 21:17 |
| clarkb | we tried to but it didn't work | 21:17 |
| *** CaptTofu has quit IRC | 21:17 | |
| sdague | yeh, the screen races | 21:17 |
| lifeless | 3:05:17 is in the middle of | 21:17 |
| lifeless | 2014-01-31 03:05:15.884 | tempest.api.compute.security_groups.test_security_group_rules_negative.SecurityGroupRulesNegativeTestJSON.t | 21:17 |
| lifeless | 2014-01-31 03:05:15.884 | tempest.api.compute.security_groups.test_security_group_rules_negative.SecurityGroupRulesNegativeTestJSON.t | 21:18 |
| lifeless | stuff in the test log | 21:18 |
| lifeless | I have to run, day with C - sorry | 21:18 |
| anteaya | have a good day, lifeless | 21:18 |
| sdague | enjoy | 21:18 |
| *** russellb_ has joined #openstack-infra | 21:30 | |
| *** ociuhandu has joined #openstack-infra | 21:30 | |
| *** _david_ has joined #openstack-infra | 21:30 | |
| *** VijayT has quit IRC | 21:32 | |
| *** gokrokve has joined #openstack-infra | 21:33 | |
| *** mgagne has quit IRC | 21:35 | |
| devananda | 19:17:35 < clarkb> you put them in devstack, then you flip the bit in devstack gate. Since devstack gate is self testing that will gate itself on the ability to test the nova-ironic driver | 21:39 |
| devananda | clarkb: question on that ... | 21:39 |
| clarkb | my devstack patch did not fix guestfs file injection so :( | 21:39 |
| clarkb | devananda: sure | 21:39 |
| devananda | clarkb: how does devstack-gate being self-testing solve the problem of "the patch hasn't landed in nova yet" | 21:39 |
| clarkb | devananda: it doesn't you ahve to land it in nova first | 21:39 |
| devananda | ah | 21:39 |
| devananda | so nova has to accept a driver without tests :) | 21:39 |
| clarkb | but you don't actually gate on it until the d-g change self gates | 21:39 |
| clarkb | devananda: no they accept it with unittests and the like | 21:40 |
| devananda | well, right | 21:40 |
| clarkb | but without integration testing for a short period yes | 21:40 |
| devananda | k | 21:40 |
| *** gokrokve has quit IRC | 21:40 | |
| devananda | *without functional tests | 21:40 |
| devananda | thanks | 21:40 |
| *** CaptTofu has joined #openstack-infra | 21:41 | |
| anteaya | clarkb: boo, what might be the next step to fix guestfs? | 21:44 |
| anteaya | any chance superbowl might fix it? | 21:45 |
| russellb_ | just disable it for now IMO | 21:45 |
| * russellb_ blames ubuntu's old guestfs version :-p | 21:45 | |
| clarkb | russellb_: the guestfs docs say it should work if I do the things I did | 21:46 |
| russellb_ | but i'm not going to have time to look at it in detail for a while ... | 21:46 |
| clarkb | apparnetly not | 21:46 |
| russellb_ | later this week i can get the guestfs author to help debug it if needed | 21:46 |
| anteaya | is that saucy image still available? | 21:46 |
| russellb_ | anteaya: that only updated the kernel | 21:47 |
| anteaya | would it be worthwhile testing on it? | 21:47 |
| anteaya | ah | 21:47 |
| russellb_ | doubtful | 21:47 |
| anteaya | k | 21:47 |
| clarkb | anteaya: it is still available but ya I don't think it will help | 21:47 |
| anteaya | should we get rid of that image? | 21:47 |
| anteaya | for the sake of tidy? | 21:47 |
| clarkb | no, not yet. I secretly want to poke at it more | 21:47 |
| anteaya | k | 21:47 |
| clarkb | curious if we flip the ext4 bit if 3.2 vs 3.11 makes a difference | 21:48 |
| devananda | clarkb: fwiw, summary of ironic's test and dependency status just sent to ML. much of it comes from // is relevant to the conversation we just had | 21:48 |
| devananda | russellb_: that email is probably also of interest to you | 21:48 |
| russellb_ | devananda: cool | 21:48 |
| anteaya | http://libguestfs.org/ mentions we are using it | 21:50 |
| *** sarob has joined #openstack-infra | 21:53 | |
| *** sarob has quit IRC | 21:58 | |
| *** cody-somerville has joined #openstack-infra | 22:00 | |
| *** e0ne has joined #openstack-infra | 22:02 | |
| sdague | clarkb: so have you proposed a patch to devstack to default ephemeral to ext4? | 22:03 |
| *** e0ne has quit IRC | 22:03 | |
| *** CaptTofu has quit IRC | 22:04 | |
| russellb_ | sdague: we can't win with guestfs apparently | 22:05 |
| sdague | yeh, though we are still running ext3 on guests... which I wonder if that's part of it | 22:05 |
| russellb_ | worth a shot | 22:05 |
| sdague | and I'd forgotten that reverted | 22:05 |
| sdague | ok, proposing something now | 22:06 |
| clarkb | sdague: no | 22:06 |
| *** thuc has joined #openstack-infra | 22:06 | |
| sdague | realistically, defaulting ephemeral to ext4 in devstack is a good idea anyway | 22:07 |
| clarkb | I do have a change up to make the underlying fs mount for the slaves ext4, btu the guests need it too | 22:07 |
| clarkb | sdague: realistically for all of openstack | 22:07 |
| clarkb | but apparently only if we can do so with a idfferent file name or something | 22:07 |
| clarkb | also SUSE | 22:07 |
| sdague | yeh, just silly cloud providers have really old guests they support ;) | 22:07 |
| sdague | I think I merged the warning that that will change | 22:07 |
| sdague | https://review.openstack.org/#/c/70555/ | 22:08 |
| sdague | clarkb: any idea why it seems to take 30 minutes to allocate d-g nodes even when we are kind of idle? | 22:11 |
| *** thuc has quit IRC | 22:12 | |
| sdague | I'd have expected them to pop from the pool quicker | 22:12 |
| clarkb | sdague: I think nodepool thinks it has available nodes when it doesn't | 22:13 |
| sdague | clarkb: ok, interestin | 22:13 |
| clarkb | I can poke at that later, but now I need to run errands | 22:13 |
| sdague | yep, enjoy | 22:13 |
| *** boris-42 has quit IRC | 22:19 | |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Add query for bug 1265495 https://review.openstack.org/70556 | 22:26 |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 22:26 | |
| *** mgagne has joined #openstack-infra | 22:26 | |
| *** russellb_ has quit IRC | 22:38 | |
| *** gokrokve has joined #openstack-infra | 22:51 | |
| *** sarob has joined #openstack-infra | 22:53 | |
| *** gokrokve has quit IRC | 22:55 | |
| *** sarob has quit IRC | 22:58 | |
| *** masayukig has joined #openstack-infra | 23:00 | |
| *** morganfainberg|z is now known as morganfainberg | 23:00 | |
| *** e0ne has joined #openstack-infra | 23:02 | |
| *** flaper87 is now known as flaper87|afk | 23:02 | |
| *** e0ne has quit IRC | 23:05 | |
| *** _david_ has quit IRC | 23:06 | |
| *** SergeyLukjanov_ is now known as SergeyLukjanov | 23:12 | |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack-infra/elastic-recheck: Add query for grenade/swift bug 1274539 https://review.openstack.org/70130 | 23:15 |
| *** masayukig has quit IRC | 23:26 | |
| *** masayukig has joined #openstack-infra | 23:26 | |
| *** dstanek_afk is now known as dstanek | 23:33 | |
| *** dims has quit IRC | 23:38 | |
| *** dims has joined #openstack-infra | 23:39 | |
| *** starmer has quit IRC | 23:44 | |
| *** dims has quit IRC | 23:45 | |
| *** dims has joined #openstack-infra | 23:47 | |
| *** starmer has joined #openstack-infra | 23:51 | |
| *** sarob has joined #openstack-infra | 23:53 | |
| *** SergeyLukjanov is now known as SergeyLukjanov_ | 23:55 | |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 23:56 | |
| *** sarob has quit IRC | 23:58 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!