| *** thorst has joined #openstack-powervm | 00:04 | |
| *** edmondsw has joined #openstack-powervm | 00:30 | |
| *** thorst has quit IRC | 00:31 | |
| *** edmondsw has quit IRC | 00:34 | |
| *** thorst has joined #openstack-powervm | 01:08 | |
| *** thorst has quit IRC | 01:08 | |
| *** thorst has joined #openstack-powervm | 01:09 | |
| *** thorst has quit IRC | 01:13 | |
| *** svenkat has joined #openstack-powervm | 01:21 | |
| *** thorst has joined #openstack-powervm | 01:25 | |
| *** thorst has quit IRC | 01:27 | |
| *** svenkat has quit IRC | 02:23 | |
| *** thorst has joined #openstack-powervm | 02:28 | |
| *** thorst has quit IRC | 02:29 | |
| *** chhavi has quit IRC | 02:33 | |
| *** esberglu has quit IRC | 02:40 | |
| *** thorst has joined #openstack-powervm | 04:04 | |
| *** thorst has quit IRC | 04:09 | |
| *** chhavi has joined #openstack-powervm | 04:13 | |
| *** thorst has joined #openstack-powervm | 04:26 | |
| *** thorst has quit IRC | 04:30 | |
| *** k0da has joined #openstack-powervm | 06:11 | |
| *** thorst has joined #openstack-powervm | 06:26 | |
| *** thorst has quit IRC | 06:31 | |
| *** k0da has quit IRC | 07:11 | |
| *** thorst has joined #openstack-powervm | 08:27 | |
| *** thorst has quit IRC | 08:32 | |
| *** svenkat has joined #openstack-powervm | 09:06 | |
| *** svenkat has quit IRC | 09:43 | |
| *** apearson has joined #openstack-powervm | 09:49 | |
| *** thorst has joined #openstack-powervm | 10:28 | |
| *** svenkat has joined #openstack-powervm | 10:30 | |
| *** svenkat has quit IRC | 10:31 | |
| *** thorst has quit IRC | 10:33 | |
| *** apearson has quit IRC | 10:46 | |
| *** smatzek has joined #openstack-powervm | 11:49 | |
| *** svenkat has joined #openstack-powervm | 11:54 | |
| *** esberglu has joined #openstack-powervm | 11:59 | |
| *** svenkat_ has joined #openstack-powervm | 12:00 | |
| *** svenkat has quit IRC | 12:01 | |
| *** svenkat_ is now known as svenkat | 12:01 | |
| *** thorst has joined #openstack-powervm | 12:02 | |
| *** esberglu has quit IRC | 12:04 | |
| *** edmondsw has joined #openstack-powervm | 12:27 | |
| *** esberglu has joined #openstack-powervm | 13:00 | |
| efried | esberglu See the dev ML - tonyb is asking for help with our -powervm branches wrt requirements thaw. | 13:12 |
|---|---|---|
| *** kylek3h has joined #openstack-powervm | 13:15 | |
| *** k0da has joined #openstack-powervm | 13:57 | |
| esberglu | efried: Talking about it in #openstack-release. You know if our repos are under TC governance? | 13:58 |
| esberglu | edmondsw: thorst: ^ | 13:58 |
| efried | They're not. | 13:58 |
| efried | Joining. | 13:58 |
| thorst | adreznec: you want to participate in that? | 13:58 |
| adreznec | Joining | 13:59 |
| esberglu | efried: edmondsw: http://184.172.12.213/89/491589/3/check/nova-out-of-tree-pvm/3315142/ | 14:28 |
| esberglu | Seeing some runs like ^ | 14:28 |
| esberglu | where the compute isn't starting up properly | 14:28 |
| esberglu | With Internal Server Error 500 | 14:29 |
| edmondsw | looking | 14:29 |
| efried | esberglu I'll say this again: if we're only failing 56 tests when the compute service doesn't start at all... do we need the other 718 at all? | 14:31 |
| edmondsw | :) | 14:31 |
| edmondsw | most likely not | 14:32 |
| efried | Here's another thing that's been on my mind. | 14:32 |
| efried | Other CIs, when you click a .log.gz, it opens in the browser, and even has convenient color coding and log level filters. | 14:32 |
| efried | Ours.... download. | 14:32 |
| edmondsw | I wonder why that is | 14:33 |
| efried | esberglu Can you add to the wishlist to figure out why that is and how we can conform? | 14:33 |
| efried | Cause that's way more convenient than save, unzip, page. | 14:33 |
| esberglu | I think it's just txt.gz vs. log.gz | 14:34 |
| esberglu | I can probably just rename them | 14:34 |
| efried | esberglu That compute log looks truncated. | 14:34 |
| esberglu | efried: Why do you say that? It goes from the service starting to the compute driver being shut down | 14:36 |
| efried | errumuh... let me download it again. | 14:37 |
| edmondsw | efried when I open n-cpu.log with less, it's missing several lines at the top that are there when I open with vi... | 14:37 |
| edmondsw | some kind of formatting issue? | 14:37 |
| efried | oh, well wtf? | 14:37 |
| esberglu | weird | 14:38 |
| edmondsw | and moving around the file in less I'm seeing weird behavior, other lines go missing and then come back... | 14:38 |
| efried | yeahno, I'm not seeing any difference. less, vi, gvim... | 14:38 |
| efried | They all start with -- Logs begin at Thu 2017-08-10 07:07:45 CDT, end at Thu 2017-08-10 07:45:23 CDT. -- | 14:39 |
| efried | Aug 10 07:30:39.694738 powervm-ci-powervm-devstacked-33602 systemd[1]: Started Devstack devstack@n-cpu.service. | 14:39 |
| efried | and end with Aug 10 07:34:58.689495 powervm-ci-powervm-devstacked-33602 nova-compute[29612]: <ServerAdapter kxe="false" kb="CUR" schemaVersion="V1_6_0"> | 14:39 |
| efried | Aug 10 07:34:58.689694 powervm-ci-powervm-devstacked-33602 nova-compute[29612]: <Metadata> | 14:39 |
| edmondsw | yeah, that's what I see with vi | 14:39 |
| edmondsw | with less I'm seeing it start with this: | 14:39 |
| edmondsw | Aug 10 07:30:51.054148 powervm-ci-powervm-devstacked-33602 nova-compute[29612]: any related configuration options. Users of neutron, meanwhile, should use the | 14:39 |
| edmondsw | maybe it's something with my mac | 14:39 |
| efried | 1551 lines. | 14:39 |
| esberglu | Nope same for me I had a different n-cpu log open | 14:40 |
| efried | okay phew. | 14:40 |
| efried | World got really weird there for a sec. | 14:40 |
| edmondsw | still weird for me :) | 14:40 |
| efried | I mean, that's still weird, but... | 14:40 |
| *** tjakobs has joined #openstack-powervm | 14:40 | |
| efried | edmondsw Are you using `less -R` ? | 14:40 |
| edmondsw | someone else's problem :) | 14:40 |
| edmondsw | yeah | 14:40 |
| efried | 227495 bytes unzipped | 14:41 |
| efried | What does wc -l give you? | 14:41 |
| edmondsw | don't have the problem with less if I remove the -r | 14:41 |
| efried | hm | 14:41 |
| edmondsw | 1551, but that wouldn't matter | 14:41 |
| efried | edmondsw -r or -R? | 14:41 |
| edmondsw | -r | 14:42 |
| efried | They're different (opposite, almost, I think) | 14:42 |
| edmondsw | hmm... -R I don't see the problem... | 14:42 |
| efried | ah, yes, I get the same bizarre behavior with -r | 14:42 |
| edmondsw | ok, I'll use -R... tx :) | 14:43 |
| efried | And forget what I said about opposite. -R is -r on roids. | 14:43 |
| edmondsw | 'Unlike -r, the screen appearance is maintained correctly in most cases.' | 14:43 |
| *** tjakobs has quit IRC | 14:44 | |
| esberglu | edmondsw: efried: Yeah the log is getting cut off in some runs | 14:46 |
| efried | Is this a fs space issue again? | 14:46 |
| efried | or possibly a not-waiting-for-sync issue? | 14:47 |
| esberglu | I miss screen... | 14:47 |
| esberglu | I don't think it's a space issue | 14:47 |
| edmondsw | esberglu are you grabbing the logs from disk, or running a journalctl command to generate them? | 14:49 |
| esberglu | journalctl command to output them to files then the files get copied to the logserver | 14:49 |
| edmondsw | hmm | 14:49 |
| esberglu | The only time I'm seeing this is on these runs where the compute driver is failing to start | 14:50 |
| edmondsw | you wouldn't expect journalctl to have a sync issue | 14:50 |
| esberglu | However ALL of the logs could be getting cut off, there isn't a good reference point for the passing runs | 14:50 |
| efried | esberglu Welll, we could shut down the service(s) before we collect the logs. | 14:52 |
| efried | Then we'd have a reference point. | 14:53 |
| esberglu | efried: edmondsw: 5680 | 15:01 |
| efried | esberglu Why not as part of process_logs? | 15:02 |
| *** k0da has quit IRC | 15:03 | |
| esberglu | Idk it's not really doing any log processing. I guess you could argue it's setting up for log processing? | 15:03 |
| esberglu | Actually it can't be part of process logs as we have it right now | 15:04 |
| esberglu | Because it has to happen before the journalctl output to files | 15:04 |
| esberglu | (now that could probably be moved to process_logs) | 15:05 |
| esberglu | efried: See ps2 | 15:09 |
| efried | esberglu Like it. Anything that reduces shell-code-in-yaml. | 15:11 |
| esberglu | efried: One more change in ps3. In browser logs | 15:21 |
| efried | esberglu Did you try that out? Does it work? | 15:22 |
| efried | .txt.gz? | 15:22 |
| efried | Honestly, I kinda doubt it's gonna have the fancy log filtering stuff. | 15:22 |
| esberglu | Well it has to be txt.gz if you want to view in the browser | 15:23 |
| efried | good enough for now. | 15:23 |
| esberglu | efried: http://184.172.12.213/manual/rest_logs/logs/n-cpu.txt.gz | 15:23 |
| esberglu | Thats what it will look like | 15:23 |
| efried | Hm, yeah, the filtering stuff is coming through, but not the coloring. Wonder what's up with that. | 15:24 |
| esberglu | Not the cleanest look, but in browser | 15:24 |
| edmondsw | yep, better | 15:24 |
| *** k0da has joined #openstack-powervm | 15:24 | |
| efried | esberglu It also doesn't actually appear to be gzipped. | 15:25 |
| efried | ...when I download it. | 15:25 |
| edmondsw | that probably depends on how you download it, right? | 15:26 |
| esberglu | What are you viewing it with? Some tools automatically handle txt.gz | 15:26 |
| esberglu | Check the file size | 15:26 |
| esberglu | Yeah it's definitely zipped. 405K. Unzipped it's 4.4M | 15:27 |
| efried | I was trying that, but you pulled the rug. | 15:27 |
| esberglu | efried: Should be good now | 15:27 |
| efried | Well, I guess my browser (?) is unzipping it automatically | 15:28 |
| efried | But not renaming it. | 15:28 |
| efried | Weird. | 15:28 |
| esberglu | efried: Yeah that's why log.gz didn't work. The browser doesn't know how to handle it, but for txt.gz it does | 15:29 |
| efried | Anyway, yeah, would like to get that coloring figured out too. I'd rather suffer the inconvenience of downloading than look at it in b/w with the color code gorp. | 15:29 |
| efried | But let's not get distracted. | 15:29 |
| esberglu | efried: This is going to take a while to get into CI, image template rebuild and all that jazz | 15:30 |
| efried | oh, boo. | 15:30 |
| efried | then we might as well get distracted, if it's going to be fairly easy to figure out. | 15:30 |
| esberglu | efried: And we can't make process_logs external to take advantage of the immediate changes like we can with other things | 15:31 |
| efried | Because IPs? | 15:32 |
| esberglu | Yep | 15:32 |
| esberglu | efried: Does the coloring still work when you download? | 15:33 |
| efried | esberglu Yes | 15:34 |
| esberglu | This might just be a browser issue that we can't get around then | 15:34 |
| efried | That doesn't make sense, because in the same browser for other CIs, it shows up colored. | 15:35 |
| esberglu | Oh okay didn't realize that | 15:35 |
| efried | http://logs.openstack.org/77/488177/6/check/gate-tempest-dsvm-ironic-ipa-partition-redfish-tinyipa-ubuntu-xenial/5e6c5b6/logs/screen-ir-api.txt.gz | 15:35 |
| efried | esberglu How does it show up for you? | 15:36 |
| esberglu | efried: Yeah thats colored | 15:36 |
| efried | I just picked that one at random, btw. They're all like that. | 15:36 |
| efried | Hum, if I look at the source, it's truly HTMLified. I wonder if they're putting it through a postprocessor. | 15:36 |
| efried | esberglu But you have a way to look at other orgs' CIs, right? | 15:37 |
| esberglu | efried: Yeah | 15:37 |
| efried | esberglu Nyak, when I look at yours ( view-source:http://184.172.12.213/manual/rest_logs/logs/n-cpu.txt.gz ) it's also HTML - just the classes are empty and the color codes aren't stripped. | 15:38 |
| edmondsw | compare the raw file contents when downloading the gz file and extracting locally (not having the browser do it) and see what's different | 15:54 |
| efried | How would I download the GZ file? curl? | 15:55 |
| efried | I can't even get curl --raw to stop decompressing it. | 15:58 |
| *** dwayne has quit IRC | 16:34 | |
| edmondsw | efried go to the parent directory, right click the link to the gz file and "save link as" | 16:58 |
| efried | edmondsw When I do that, it unzips it for me anyway. | 16:58 |
| edmondsw | boo | 16:58 |
| efried | (but doesn't rename it) | 16:58 |
| edmondsw | wget? | 16:59 |
| efried | edmondsw Still unzips it. | 16:59 |
| edmondsw | then you're probably stuck | 16:59 |
| efried | In fact, it may even be unzipping it on the server. | 16:59 |
| edmondsw | it would have to be | 17:00 |
| efried | which is weird. Like, what's the point of having zippable content? | 17:00 |
| edmondsw | saves disk space, just not network bandwidth | 17:05 |
| edmondsw | I'm guessing browsers don't have the builtin support for rendering these files that we assumed they did, so they do it server-side to make it work | 17:06 |
| efried | edmondsw Well, what's weird is, we're not doing any of that rendering on the server side, but *some* of it is happening anyway. | 17:11 |
| efried | Hm, if it's server side... I wonder if we just have an outdated apache or something on our log server. | 17:12 |
| efried | or missing a plugin | 17:12 |
| edmondsw | esberglu that sed command you've got doesn't seem right... | 17:19 |
| edmondsw | in 5681 | 17:19 |
| efried | edmondsw What doesn't work? | 17:21 |
| edmondsw | the command :) | 17:21 |
| edmondsw | I don't ever get cleared... | 17:22 |
| edmondsw | I would also think we'd need to keep the first part of the hostname so we know what host it is | 17:22 |
| edmondsw | esberglu efried figured out a command that would work and commented with that | 17:30 |
| efried | esberglu See -nova for answer on the log coloring business. | 17:31 |
| edmondsw | that doesn't read well... meant to say "both of you... I figured out..." | 17:31 |
| esberglu | edmondsw: Yep that seems to work | 17:32 |
| esberglu | Waiting for my manual run to finish so I can test and then we can pull the trigger | 17:33 |
| efried | edmondsw How does it help us to keep the first part of the domain name? | 17:34 |
| edmondsw | we were before... I assumed that was how we were figuring out which host was used | 17:34 |
| edmondsw | one problem with my proposal is if we ever print just the domain info, not a full hostname, it wouldn't touch that | 17:35 |
| edmondsw | do we do that? | 17:35 |
| esberglu | edmondsw: Not to my knowledge | 17:35 |
| esberglu | I will make sure that everything gets cleared in test | 17:36 |
| esberglu | efried: The first part of the domain name stays | 17:36 |
| efried | fine fine. | 17:37 |
| esberglu | edmondsw: Is that 1 in 1.cleared.domain.name getting substituted somehow? | 17:37 |
| edmondsw | \1 | 17:38 |
| edmondsw | means the first () match | 17:38 |
| edmondsw | that's how it keeps the first part of the hostname | 17:38 |
| esberglu | efried: Catching up on the nova convo | 17:44 |
| edmondsw | esberglu sorry, that sed command I recommended needs another tweak... commented | 17:47 |
| esberglu | edmondsw: np. Done | 17:50 |
| esberglu | efried: Sounds like it's time to get this mod_wsgi, uwsgi business figured out | 17:51 |
| esberglu | You're stacking with that change again? | 17:51 |
| efried | esberglu Working on it in -nova | 17:51 |
| efried | yes | 17:52 |
| efried | esberglu It kinda looked like /tmp ran out of space. | 17:52 |
| efried | While creating the image | 17:52 |
| efried | Speculation is that under uwsgi, we copy the whole image through /tmp | 17:52 |
| efried | which is... silly. | 17:53 |
| edmondsw | nasty | 17:54 |
| *** chhavi has quit IRC | 18:08 | |
| *** dwayne has joined #openstack-powervm | 18:12 | |
| *** smatzek has quit IRC | 18:32 | |
| esberglu | edmondsw: efried: Thought I was merging the neo-os-ci change, actually merged the powervm-ci change. Not an issue, just means that one more changeset will be going up | 18:36 |
| *** smatzek has joined #openstack-powervm | 18:36 | |
| esberglu | edmondsw: efried: 5682 | 18:40 |
| efried | esberglu Can you copy out /var/log/apache2/* in that same patch? | 18:45 |
| esberglu | efried: Sure | 18:45 |
| efried | esberglu Should be moot when we get uwsgi figured out... which looks like it's getting close. | 18:45 |
| esberglu | efried: Actually that would have to go in the jenkins job config. I can still do it in a separate patch if you would like | 18:47 |
| esberglu | Or is it not worth spending time on? | 18:48 |
| efried | Huh? You can't add [ for f in `ls /var/log/apache2/*.log`; do cp $f /opt/stack/logs/$f.txt; done ] in that same script? | 18:49 |
| esberglu | efried: Yeah brain fart | 18:50 |
| esberglu | efried: http://paste.openstack.org/show/618090/ | 18:56 |
| esberglu | Any idea why that second command isn't picking up the log files? | 18:56 |
| efried | I wonder if sudo refuses to expand wildcards | 18:57 |
| efried | you know, for safety. | 18:57 |
| efried | esberglu Anyway, just copy everything out of there. | 18:57 |
| *** efried is now known as efried_afk | 19:02 | |
| *** openstackgerrit has quit IRC | 19:03 | |
| *** efried_afk is now known as efried | 19:27 | |
| efried | esberglu uwsgi might be fixed by https://review.openstack.org/#/c/492651 -- trying now. | 19:33 |
| esberglu | efried: Nice | 19:44 |
| esberglu | efried: edmondsw: thorst: Starting the process to tag RC1 and create the pike branches. For anything to get in after that point it will have to be backported to pike | 19:56 |
| efried | esberglu Go. | 19:56 |
| edmondsw | +1 | 19:57 |
| *** smatzek has quit IRC | 20:07 | |
| esberglu | Pike branches are all up | 20:13 |
| esberglu | efried: Did your stack work? | 20:17 |
| efried | esberglu No. But it may be because I had stale settings in my apache confs. Cleared 'em out, trying again. | 20:17 |
| efried | esberglu Do we have a devstack bug open for the stupid neutron conf dir bs? | 20:18 |
| thorst | esberglu: cool :-) | 20:18 |
| esberglu | efried: no | 20:21 |
| efried | esberglu I can't imagine why we're the only ones seeing this. | 20:22 |
| esberglu | efried: When I first started hitting it I thought it was an issue on our end for that reason | 20:22 |
| esberglu | But not convinced of that anymore | 20:22 |
| *** esberglu has quit IRC | 20:47 | |
| edmondsw | did we really never open a bug for that? looking... | 20:52 |
| edmondsw | efried esberglu I did open a bug: https://bugs.launchpad.net/devstack/+bug/1699609 | 20:55 |
| openstack | Launchpad bug 1699609 in devstack "stack fails unable to find /etc/neutron/neutron.conf" [Undecided,New] | 20:55 |
| *** k0da has quit IRC | 20:59 | |
| *** esberglu has joined #openstack-powervm | 21:01 | |
| *** esberglu_ has joined #openstack-powervm | 21:02 | |
| *** esberglu has quit IRC | 21:02 | |
| *** thorst has quit IRC | 21:24 | |
| *** thorst has joined #openstack-powervm | 21:27 | |
| efried | esberglu_ edmondsw Workaround seems to be removing the qos service. Do we need that for anything? | 21:27 |
| *** thorst has quit IRC | 21:31 | |
| esberglu_ | I don't think so | 21:37 |
| efried | esberglu_ Let's kill it and see if we get any additional failures. | 21:40 |
| *** thorst has joined #openstack-powervm | 21:49 | |
| *** thorst has quit IRC | 21:54 | |
| *** svenkat has quit IRC | 21:59 | |
| *** kylek3h has quit IRC | 22:07 | |
| *** openstackgerrit has joined #openstack-powervm | 22:12 | |
| openstackgerrit | Matthew Edmonds proposed openstack/nova-powervm master: ISCSI volume attachment fixes https://review.openstack.org/467599 | 22:12 |
| *** openstackgerrit has quit IRC | 22:18 | |
| edmondsw | efried esberglu I wonder if the fact that we're now using a generated tempest conf will help us remove qos... | 22:20 |
| edmondsw | I also wonder if that will really fix this, or if we'll just find that another plugin that we do need hits the same issue | 22:23 |
| edmondsw | doesn't look like it, though... qos seems to be the only thing we've got turned on from these: https://github.com/openstack/neutron/blob/master/devstack/plugin.sh#L21-L43 | 22:30 |
| edmondsw | so yeah, if we could drop qos, that would probably get rid of the problem for us | 22:30 |
| efried | edmondsw Well, I can confirm that stack goes through with /etc/neutron missing and qos off the enabled list. | 22:34 |
| edmondsw | cool | 22:35 |
| edmondsw | we should check with thorst to see if he knows of a reason we need it | 22:35 |
| *** esberglu_ has quit IRC | 22:47 | |
| *** edmondsw has quit IRC | 22:53 | |
| *** thorst has joined #openstack-powervm | 23:38 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!