| greghaynes | yep, looks like they call it i686 | 00:00 |
|---|---|---|
| *** CaptTofu has quit IRC | 00:00 | |
| *** CaptTofu has joined #tripleo | 00:00 | |
| clarkb | are you guys still using i386 for memory constraints? | 00:00 |
| Lotus907efi | clarkb: before yu completely dismiss the httpretty bug see the note at the bottom of the bug report please | 00:00 |
| clarkb | Lotus907efi: which note? | 00:01 |
| * greghaynes wonders if lifeless will not experience the OOM on x86 also | 00:01 | |
| Lotus907efi | once httpretty fails to build the command that executes afterwards takes f'in FOREVER to complete | 00:01 |
| clarkb | Lotus907efi: right it is continuing on and do all of the othe rwheels | 00:01 |
| clarkb | Lotus907efi: at least that is what I would expect it to do. which means compiling stuff | 00:01 |
| Lotus907efi | yes, but should the next command take 10 - 15 minutes for ONE thing? | 00:01 |
| clarkb | Lotus907efi: you can check by seeing if you get any wheels from packages listed after httpretty | 00:01 |
| clarkb | Lotus907efi: sure its literally compiling all of the requirements so that it can build them into wheels | 00:02 |
| jogo | lifeless: what version of libvirt do you guys use again? | 00:02 |
| Lotus907efi | on a 24 core machine with 24GB of memory? | 00:02 |
| jogo | lifeless: this is about which version of libvirt to test in devstack-gate | 00:02 |
| clarkb | Lotus907efi: sure, it doesn't necessarily do the compiles in parallel | 00:03 |
| Lotus907efi | hmmm, ok | 00:03 |
| *** andreaf has joined #tripleo | 00:03 | |
| Lotus907efi | clarkb: I just checked two packages that occur AFTER httpretty | 00:04 |
| Lotus907efi | no whl files | 00:04 |
| Lotus907efi | let me continue checking | 00:05 |
| Lotus907efi | hmm, out of 6 - 8 things after httpretty only one I have checked has a whl file | 00:06 |
| Lotus907efi | boto | 00:06 |
| openstackgerrit | Gregory Haynes proposed a change to openstack/tripleo-image-elements: Use xtradb-cluster in place of mysql https://review.openstack.org/75540 | 00:07 |
| clarkb | Lotus907efi: ok I spent a bit more time reading the code. pypi-mirror seems to do a best effort with the wheel command | 00:07 |
| clarkb | Lotus907efi: it then does the tarballs and only that return code is checked. So it should continue on just fine | 00:07 |
| clarkb | but not everything may get a wheel | 00:07 |
| Lotus907efi | hmmm, ok | 00:08 |
| Lotus907efi | so what was the answer to my question about pip having a whl file but no tar.gz file? | 00:09 |
| clarkb | Lotus907efi: https://git.openstack.org/cgit/openstack-infra/pypi-mirror/tree/pypi_mirror/cmd/run_mirror.py#n294 what happens there is it says give me a list of everyting I did manage to install now let us get the tarball and wheels for those things | 00:09 |
| clarkb | the wheel step may fail again | 00:09 |
| Lotus907efi | I know you said pip and some other things were treated as special cases | 00:09 |
| Lotus907efi | due to dependencies | 00:09 |
| clarkb | Lotus907efi: yes see scrollback above where I like to the code | 00:09 |
| Lotus907efi | the bit about upgrade_format and the no-use-wheel thing? | 00:11 |
| clarkb | yes | 00:11 |
| Lotus907efi | so for special cases like pip it would NOT use a whl file and make sure the tar.gz file was pulled down | 00:12 |
| lifeless | jogo: saucies | 00:13 |
| lifeless | jogo: today | 00:13 |
| lifeless | greghaynes: re'll find out :) | 00:14 |
| clarkb | Lotus907efi: current code appears to just use a wheel | 00:14 |
| clarkb | Lotus907efi: so I think the httpretty bug is invalid but the pip bug is valid | 00:14 |
| lifeless | clarkb: our CI servers do amd64 | 00:14 |
| Lotus907efi | hmm, ok | 00:14 |
| Lotus907efi | sounds right | 00:14 |
| greghaynes | lifeless: I just pushed what *should* fix it | 00:14 |
| greghaynes | gotta wait till my devtest with another patch finishes | 00:14 |
| jogo | lifeless: thanks that would be 1.1.1 | 00:15 |
| Lotus907efi | greghaynes: how long do your devtest.sh runs take on average? and are you using internal mirrors? | 00:15 |
| greghaynes | Lotus907efi: mine are about 30ish mins | 00:15 |
| greghaynes | and yes, mirrors | 00:15 |
| Lotus907efi | wow, that is pretty fast | 00:15 |
| greghaynes | raid0 ssd with noatime for great good | 00:16 |
| Lotus907efi | can I speed this up even more if we parallelize the code on a HPC cluster? | 00:16 |
| Lotus907efi | I might have 1500 nodes hanging around in a few weeks :) | 00:16 |
| lifeless | greghaynes: thats happier | 00:17 |
| greghaynes | *shrug* - I did hear about someone tmpfs'ing to speed it up | 00:17 |
| Lotus907efi | yeah, that would help too | 00:17 |
| Lotus907efi | we do have some BIG nodes here in hpcloud testing rack with 148GB of memory | 00:17 |
| lifeless | greghaynes: relatime should be fine | 00:17 |
| greghaynes | probably want more than 24g ram for that | 00:17 |
| greghaynes | Yea, I noatime so my ssd's (in theory) live longer | 00:17 |
| Lotus907efi | I think they are DL380 G8 systems | 00:18 |
| Lotus907efi | when I was working at Intel a few years ago they had some HPC nodes that had over 300GB of memory | 00:19 |
| greghaynes | heh | 00:19 |
| tchaypo | and then for a really fast build, make sure you put your swap file on tmpfs too | 00:19 |
| Lotus907efi | hehehehe | 00:19 |
| greghaynes | my deal is just 32gb ram, but might be able to tmpfs that too | 00:19 |
| StevenK | tchaypo: Or just ignore swap | 00:20 |
| * greghaynes facepalms @ tchaypo | 00:20 | |
| greghaynes | yes, like why even have swap at that point :p | 00:20 |
| Lotus907efi | yeah, on those systems there was no swap space | 00:20 |
| *** matsuhashi has joined #tripleo | 00:20 | |
| tchaypo | thank you greghaynes :) | 00:20 |
| greghaynes | hehe | 00:20 |
| tchaypo | StevenK: ah, you're awake! | 00:20 |
| Lotus907efi | no, this version of StevenK is just a dream within a dream | 00:21 |
| StevenK | tchaypo: Have been for a few hours? | 00:21 |
| lifeless | greghaynes: you have swap space? | 00:23 |
| * greghaynes looks | 00:23 | |
| greghaynes | yes :( | 00:23 |
| StevenK | I also have swap space on my laptop | 00:23 |
| StevenK | Which I think I might kill | 00:24 |
| greghaynes | Well, theres really no excuse - this is box dedicated to basically running devtest :p | 00:24 |
| Lotus907efi | I think my swap space is pointed at /dev/null | 00:24 |
| Lotus907efi | my system really speeded up when I did that | 00:24 |
| StevenK | Lotus907efi: Hm, swapping pages back into RAM must be fun, then. | 00:25 |
| greghaynes | swap as a replacement for memset(0, ...) | 00:25 |
| Lotus907efi | in the 1930's when an upitty customer complained to Ettore Bugatti about the brakes on one of his cars, he replied that brakes just slow you down | 00:26 |
| clarkb | Lotus907efi: ok pypi-mirror bugs updated. Feel free to take a shot at fixing the one for pip and wheels | 00:27 |
| Lotus907efi | when another customer complained about his car being hard to start when cold Bugatti said if he couldn't afford a heated garage then he shouldn't buy a Bugatti | 00:27 |
| *** CaptTofu has quit IRC | 00:27 | |
| lifeless | greghaynes: so | 00:28 |
| lifeless | 3406 mysql 20 0 827m 104m 6820 S 0.3 5.2 0:01.21 mysqld | 00:28 |
| lifeless | greghaynes: is that what you're seeing? | 00:28 |
| lifeless | greghaynes: when does the memory bloat kick in and break the world ? | 00:28 |
| greghaynes | No. For me it kicked in while it setup the undercloud (os-c-c run)... I wonder if your running 32bit makes a big difference though | 00:29 |
| lifeless | ok, odd though as o-c-c runs on seed too | 00:30 |
| lifeless | but specifically one doesn't need to build an overcloud to reproduce? | 00:30 |
| greghaynes | yep, seed didnt oom for me though for who knows why | 00:30 |
| * greghaynes kicks of devtest run with xtradb patch | 00:30 | |
| *** rpodolyaka1 has joined #tripleo | 00:40 | |
| *** andreaf has quit IRC | 00:41 | |
| Lotus907efi | so if you switch to xtradb won't that sort of invalidate all of the work and patches Thersa MChale has been submitting against mysql ? | 00:42 |
| greghaynes | Possibly some | 00:42 |
| greghaynes | although I sent the xtradb patch to her and shes been following it | 00:43 |
| lifeless | oh great | 00:43 |
| lifeless | *now* I trigger the infamous ext4 resize bug | 00:43 |
| StevenK | Which bug? | 00:43 |
| lifeless | I thought we merged dan's fix for it | 00:44 |
| greghaynes | I didnt realize that was still around | 00:44 |
| *** rpodolyaka1 has quit IRC | 00:44 | |
| lifeless | greghaynes: http://imgur.com/Z47nEKZ | 00:46 |
| greghaynes | yerp | 00:46 |
| *** yamahata has quit IRC | 00:47 | |
| * lifeless respins the image | 00:50 | |
| *** nosnos has joined #tripleo | 00:51 | |
| *** nosnos_ has joined #tripleo | 00:53 | |
| *** nosnos has quit IRC | 00:57 | |
| greghaynes | hrm, for some reason my $TE_DATAFILE is not valid json... | 01:01 |
| greghaynes | specifically, undersloud ssh-key has actual newlines in it | 01:02 |
| StevenK | lifeless: Can you +A https://review.openstack.org/#/c/75444/ ? It's had 2 +2's for > a week | 01:04 |
| Lotus907efi | hmm, I cannot get an undersloud to build, all I end up with are all these damn underclouds | 01:06 |
| StevenK | greghaynes: Hm. I have a patch up that uses $(cat) inside jq, and that has literal '\n's in the string | 01:06 |
| Lotus907efi | jq? I heard that mentioned somewhere else today | 01:06 |
| Lotus907efi | what is jq | 01:07 |
| greghaynes | man jq :p | 01:07 |
| Lotus907efi | is that a debian package perhaps? | 01:07 |
| greghaynes | its... interesting | 01:07 |
| Lotus907efi | ah | 01:07 |
| Lotus907efi | ok | 01:07 |
| StevenK | Lotus907efi: http://stedolan.github.io/jq/ | 01:07 |
| Lotus907efi | json preprocessor | 01:07 |
| Lotus907efi | so jq is a debian unstable package | 01:08 |
| Lotus907efi | and if some un-named "company" only allows pulls from Debian testing then this package is unavailable | 01:08 |
| lifeless | yes | 01:08 |
| lifeless | using testing is generally a terrible idea | 01:08 |
| lifeless | use unstable or stable only | 01:08 |
| lifeless | testing has nearly all the flaws of unstable but not security updates | 01:09 |
| lifeless | last I heard | 01:09 |
| Lotus907efi | well a certain new project at some here-to-for un-named company has decided to pull from debian testing | 01:09 |
| clarkb | lifeless: I heard a rumor you may have helped make a distro thing to fix that | 01:09 |
| lifeless | clarkb: Ubuntu ? | 01:10 |
| clarkb | ya :) | 01:10 |
| lifeless | fixish :P | 01:10 |
| Lotus907efi | for it's new orthogonal distro | 01:10 |
| lifeless | Lotus907efi: are you encountering an error about a missing jq ? | 01:11 |
| Lotus907efi | and since jq is not in debian testing then it is not available to test tripleo with this new orthogonal distro | 01:11 |
| Lotus907efi | I will as soon as I start testing with "orthogonal" | 01:11 |
| StevenK | It failed to build on armhf | 01:11 |
| lifeless | Lotus907efi: you should be able to just use debian unstable urls | 01:11 |
| Lotus907efi | ok, well I guess that is possible | 01:12 |
| Lotus907efi | anyway, goodnight all, I have had enough fun for one day | 01:16 |
| *** Lotus907efi is now known as Lotus907efi_ | 01:17 | |
| openstackgerrit | A change was merged to openstack/diskimage-builder: Update openSUSE package mapping for libffi-dev https://review.openstack.org/75444 | 01:18 |
| *** tserong has quit IRC | 01:33 | |
| *** rpodolyaka1 has joined #tripleo | 01:40 | |
| lifeless | oh | 01:41 |
| lifeless | I've got a new theory on this resize2fs bug | 01:41 |
| lifeless | what if the lazy metadata hasn't finished lazy updating before we unmount and freeze the image | 01:41 |
| lifeless | does resize2fs handle that... | 01:42 |
| *** tserong has joined #tripleo | 01:46 | |
| *** tserong has joined #tripleo | 01:46 | |
| *** rpodolyaka1 has quit IRC | 01:46 | |
| lifeless | greghaynes: undercloud looks fine | 01:50 |
| lifeless | 3771 mysql 20 0 828m 104m 6772 S 0.0 5.2 0:02.62 mysqld | 01:50 |
| lifeless | I'll try AMD64 | 01:50 |
| openstackgerrit | Gregory Haynes proposed a change to openstack/tripleo-image-elements: Explicitly pip install pbr before its installed as dependency https://review.openstack.org/81101 | 01:54 |
| greghaynes | hrm, and CI just had a triple SUCCESS on the xtradb patch | 01:55 |
| *** dkehn has quit IRC | 01:58 | |
| *** dkehn has joined #tripleo | 02:07 | |
| *** yamahata has joined #tripleo | 02:07 | |
| openstackgerrit | A change was merged to openstack/tripleo-incubator: Use the DIB_REPOLOCATION / DIB_REPOREF for the bootstrap. https://review.openstack.org/78371 | 02:18 |
| openstackgerrit | A change was merged to openstack/tripleo-incubator: Switch to using jq to write out initial JSON https://review.openstack.org/80884 | 02:20 |
| *** CaptTofu has joined #tripleo | 02:25 | |
| *** shakamunyi has joined #tripleo | 02:28 | |
| *** fandi has joined #tripleo | 02:33 | |
| StevenK | lifeless: Do you still have reservations about https://review.openstack.org/#/c/77741/ ? | 02:40 |
| lifeless | I never had reservations | 02:42 |
| lifeless | I just wanted to know it worked ;) | 02:42 |
| openstackgerrit | A change was merged to openstack-infra/tripleo-ci: Switch to nodes list in the JSON https://review.openstack.org/77741 | 02:42 |
| lifeless | StevenK: whats up with https://review.openstack.org/#/c/79051/ ? | 02:43 |
| *** cwolferh has quit IRC | 02:44 | |
| StevenK | lifeless: Let me rebase that | 02:47 |
| openstackgerrit | Steve Kowalik proposed a change to openstack/tripleo-incubator: Store seed details in the JSON https://review.openstack.org/79051 | 02:48 |
| StevenK | lifeless: I was plotting to finish off the remove backward compatability branch | 02:50 |
| lifeless | StevenK: cool. we'll need to deploy a new set of testenvs | 02:52 |
| StevenK | lifeless: Sure, that branch won't pass CI until we do, so that can happen whenever | 02:54 |
| lifeless | okydoky | 02:55 |
| StevenK | Sigh | 02:57 |
| StevenK | Talking to saved VMs isn't very helpful | 02:57 |
| * StevenK wants the RAM he ordered already | 02:59 | |
| StevenK | Hah, I think I can see why I can't talk to the VMs, even though they're running | 03:01 |
| lifeless | ENOROUTE | 03:01 |
| StevenK | Exactly | 03:01 |
| StevenK | Trying to work out how to resurrect that bit | 03:01 |
| lifeless | ip route add 192.0.2.0/24 via $SEEDIP | 03:02 |
| *** matsuhashi has quit IRC | 03:13 | |
| lifeless | building new te image now | 03:17 |
| *** matsuhashi has joined #tripleo | 03:20 | |
| *** matsuhashi has quit IRC | 03:25 | |
| *** lazy_prince has quit IRC | 03:28 | |
| *** xuhaiwei has quit IRC | 03:30 | |
| *** killer_prince has joined #tripleo | 03:34 | |
| lifeless | ok we have new te imagefs we can deploy | 03:35 |
| lifeless | just need to take down CI to do it ... | 03:35 |
| *** weshay has quit IRC | 03:39 | |
| *** nosnos_ has quit IRC | 03:41 | |
| *** rpodolyaka1 has joined #tripleo | 03:44 | |
| *** rpodolyaka1 has quit IRC | 03:48 | |
| *** killer_prince has quit IRC | 03:54 | |
| *** vkozhukalov_ has joined #tripleo | 03:55 | |
| openstackgerrit | Richard Su proposed a change to openstack/tripleo-image-elements: Update selinux policies for mysqld https://review.openstack.org/74607 | 03:55 |
| *** ramishra has joined #tripleo | 04:00 | |
| *** CaptTofu has quit IRC | 04:02 | |
| *** CaptTofu has joined #tripleo | 04:03 | |
| *** CaptTofu has quit IRC | 04:07 | |
| *** xuhaiwei has joined #tripleo | 04:10 | |
| *** matsuhashi has joined #tripleo | 04:23 | |
| *** nosnos has joined #tripleo | 04:35 | |
| *** rpodolyaka1 has joined #tripleo | 04:45 | |
| openstackgerrit | Steve Kowalik proposed a change to openstack-infra/tripleo-ci: Change how the JSON is updated for readability https://review.openstack.org/80905 | 04:46 |
| *** rpodolyaka1 has quit IRC | 04:49 | |
| openstackgerrit | Steve Kowalik proposed a change to openstack-infra/tripleo-ci: Populate seed.ip in the testenv JSON https://review.openstack.org/79043 | 04:51 |
| tchaypo | StevenK: https://bugs.launchpad.net/tripleo/+bug/1294083 | 04:55 |
| StevenK | Yeah, which you saw yesterday | 04:57 |
| *** gcha has quit IRC | 04:57 | |
| tchaypo | which causes me to take a look at devtest.sh to see how --existing-environment works | 04:59 |
| tchaypo | and the answer is that it skips running _testenv entirely | 04:59 |
| tchaypo | and the other fun I've just found is from | 05:01 |
| tchaypo | 13:18 < openstackgerrit> A change was merged to openstack/tripleo-incubator: Use the DIB_REPOLOCATION / DIB_REPOREF for the bootstrap. https://review.openstack.org/78371 | 05:01 |
| tchaypo | it turns out that change breaks if DIB_REPOLOCATION_diskimage_builder is not set | 05:02 |
| StevenK | Yay? | 05:03 |
| lifeless | damn | 05:05 |
| lifeless | going to revert or fix? | 05:05 |
| tchaypo | I'm going to spend about 15 minutes trying to fix, then revert | 05:05 |
| tchaypo | I'd prefer to fix but the lines I'm going to need to edit are already approaching line-noise | 05:06 |
| tchaypo | how do i test to see if a variable is set without tripping set -u? | 05:12 |
| tchaypo | I'm sure I've done this before. recently, even. | 05:12 |
| lifeless | you can use ${foo:-''} | 05:15 |
| lifeless | or the other ${NAME: operators | 05:15 |
| tchaypo | yep, just like the source-repositories element does | 05:16 |
| tchaypo | actually it's even fancier; it uses ${!REPOLOCATION_OVERRIDE:-...} | 05:19 |
| *** jesusaurus has quit IRC | 05:26 | |
| *** andreaf has joined #tripleo | 05:35 | |
| openstackgerrit | James Polley proposed a change to openstack/tripleo-incubator: Set default values for repo location and ref https://review.openstack.org/81443 | 05:37 |
| tchaypo | lifeless, StevenK - ^^ | 05:37 |
| *** xuhaiwei has quit IRC | 05:37 | |
| *** Killer_prince has joined #tripleo | 05:37 | |
| * StevenK dabs at the blood coming from his eyes | 05:38 | |
| *** tchaypo changes topic to "FIREDRILL https://review.openstack.org/81443 | tripleo-cd running preserve-ephemeral WIP patches and https://review.openstack.org/#/c/62042/ | Using OpenStack to deploy OpenStack;meetings Tuesday 1900 UTC in #openstack-meeting-alt" | 05:38 | |
| tchaypo | Does it count as a firedrill if it only breaks users and doesn't upset CI? | 05:38 |
| tchaypo | oh blah | 05:41 |
| tchaypo | did you spot the glaring mistake? | 05:41 |
| StevenK | Heh | 05:43 |
| StevenK | Yes | 05:43 |
| openstackgerrit | James Polley proposed a change to openstack/tripleo-incubator: Set default values for repo location and ref https://review.openstack.org/81443 | 05:45 |
| tchaypo | of course, it would perfectly fine when I tested it, because I don't have any of those things set so it was using the defaults. | 05:45 |
| *** rpodolyaka1 has joined #tripleo | 05:45 | |
| openstackgerrit | James Polley proposed a change to openstack/tripleo-incubator: Set default values for repo location and ref https://review.openstack.org/81443 | 05:46 |
| *** andreaf has quit IRC | 05:48 | |
| *** rpodolyaka1 has quit IRC | 05:50 | |
| tchaypo | jang: Could you review that change when you get a chance? I'd like to confirm that it still does what you intended | 05:50 |
| tchaypo | it'd be really handy if I could just set DIB_REPOREF_tripleo_incubator and have pull-tools pull down my patch | 05:51 |
| *** xuhaiwei has joined #tripleo | 05:53 | |
| tchaypo | no wait, that should actually work | 05:53 |
| tchaypo | because that will get read by devtest_setup.sh, which works fine; and by the time it runs pull-tools it will have my version | 05:54 |
| tchaypo | ah, nope. it won't re-pull the cached -incubator if it's already there | 05:59 |
| *** matsuhashi has quit IRC | 05:59 | |
| StevenK | It should update it, though, no? | 05:59 |
| tchaypo | yes | 05:59 |
| tchaypo | but the update happens *after* the reference to the unbound DIB_REPOLOCATION_diskimage_builder | 05:59 |
| lifeless | theres an interaction with ZUUL_REF too though | 05:59 |
| tchaypo | even if -incubator were the first tool in $TOOLS, by that point you're already inside the pull-tools script and it's going to die when it hits the second one | 06:00 |
| tchaypo | never mind. git review -d 81443 it is. | 06:00 |
| *** matsuhashi has joined #tripleo | 06:01 | |
| *** rpodolyaka1 has joined #tripleo | 06:01 | |
| tchaypo | http://paste.openstack.org/show/1fe6CsjEsK6K3DvwCv4t/ | 06:02 |
| tchaypo | that;s not really how I'd like devtest.sh --trash-my-machine to end | 06:03 |
| *** nati_ueno has joined #tripleo | 06:03 | |
| *** ramishra has quit IRC | 06:03 | |
| *** CaptTofu has joined #tripleo | 06:03 | |
| tchaypo | it runs | 06:06 |
| tchaypo | + git --no-pager log -1 --pretty=oneline | 06:06 |
| tchaypo | and then exits | 06:06 |
| tchaypo | exit code is 0.. why does that abort? | 06:06 |
| *** CaptTofu has quit IRC | 06:08 | |
| *** akuznetsov has quit IRC | 06:08 | |
| *** xuhaiwei has quit IRC | 06:11 | |
| *** ramishra has joined #tripleo | 06:12 | |
| lifeless | master doesn't look like a branch, or | 06:14 |
| lifeless | tchaypo: bad checking on whether to pull ;) | 06:14 |
| tchaypo | yeah, that's what I'm poking at - line 47 | 06:15 |
| tchaypo | I understand why it skips the pull, but why does that abort the whole devtest.sh execution? | 06:18 |
| lifeless | does it? the first one doesn't | 06:20 |
| *** giulivo has quit IRC | 06:21 | |
| tchaypo | oh, I see what you mean, I think | 06:22 |
| tchaypo | I'm on review/james_polley/use_default_values_for_repo_locatoin_and_ref | 06:22 |
| tchaypo | but the error message says "master doesn't look like a branch" | 06:23 |
| tchaypo | ie, it's using its default $REF even though I've manually moved it to a different branch. | 06:23 |
| lifeless | bbiaw | 06:25 |
| *** akuznetsov has joined #tripleo | 06:27 | |
| *** nati_uen_ has joined #tripleo | 06:28 | |
| openstackgerrit | James Polley proposed a change to openstack/tripleo-incubator: Set default values for repo location and ref https://review.openstack.org/81443 | 06:29 |
| *** nati_ueno has quit IRC | 06:31 | |
| *** jtomasek has joined #tripleo | 06:33 | |
| tchaypo | *facepalm* | 06:34 |
| tchaypo | the reason it exists after that point is because I was running devtest_setup.sh not devtest.sh | 06:34 |
| tchaypo | and devtest_setup.sh doesn't have any more to do | 06:34 |
| * tchaypo is so clever sometimes | 06:35 | |
| tchaypo | and that's handy, because i can now run away for dinner with a clear conscience knowing that I have a patch that works. | 06:36 |
| *** nati_uen_ has quit IRC | 06:38 | |
| *** Slower has quit IRC | 06:40 | |
| *** rlandy has joined #tripleo | 06:43 | |
| SpamapS | hm | 06:45 |
| SpamapS | so has anyone else tried to use a local mirror w/ the pypi mirror lately? | 06:45 |
| SpamapS | I got the PrettyTable fail | 06:45 |
| SpamapS | Apparently pip expects that the index url will serve things in a case-insensitive manner.. | 06:46 |
| SpamapS | but our index url is file:/// ... | 06:46 |
| SpamapS | so.. not so much | 06:46 |
| Killer_prince | SpamapS: I did try using pypi local mirror.. seems to be working with dib. but doesnt seem to work with dib when used with devtest.. | 06:47 |
| SpamapS | right | 06:47 |
| SpamapS | however, serving it via apache with CheckSpelling on does work | 06:47 |
| *** matsuhashi has quit IRC | 06:47 | |
| Killer_prince | hmm.. | 06:47 |
| SpamapS | which makes sense.. | 06:48 |
| SpamapS | but what I really want is for pip to handle the fact that file:/// is case sensitive.. | 06:49 |
| SpamapS | I think this is only showing up because we're using latest pip/virtualenv | 06:49 |
| SpamapS | mordred: ^^ | 06:49 |
| SpamapS | clarkb: ^^ | 06:49 |
| *** matsuhashi has joined #tripleo | 06:50 | |
| *** ramishra has quit IRC | 06:52 | |
| *** Slower has joined #tripleo | 06:53 | |
| *** morazi has quit IRC | 06:56 | |
| SpamapS | hm | 06:57 |
| SpamapS | building i386 images, but my local pypi mirror is x86-64 | 06:57 |
| SpamapS | seems wheels is a bit scary to actually use.. :-P | 06:57 |
| *** sseago has quit IRC | 06:58 | |
| lifeless | SpamapS: case sensitive should be fine, my mirror is http which is case sensitive | 07:00 |
| lifeless | SpamapS: wheels are quite early in learning :) | 07:02 |
| SpamapS | lifeless: if I turn off CheckSpelling, I get a fail 100% of the time | 07:03 |
| SpamapS | lifeless: or if I use the default file:///home/clint/.cache/image-create/pypi/ | 07:03 |
| StevenK | Is it still mod_speling | 07:04 |
| StevenK | ? | 07:04 |
| SpamapS | 127.0.0.1 - - [19/Mar/2014:00:00:22 -0700] "GET /pypi/mirror/Ubuntu-13.10/PrettyTable/ HTTP/1.1" 301 625 "-" "pip/1.5.4 CPython/2.7.5+ Linux/3.11.0-17-generic" | 07:04 |
| SpamapS | 127.0.0.1 - - [19/Mar/2014:00:00:22 -0700] "GET /pypi/mirror/Ubuntu-13.10/prettytable/ HTTP/1.1" 200 507 "-" "pip/1.5.4 CPython/2.7.5+ Linux/3.11.0-17-generic" | 07:04 |
| SpamapS | StevenK: yes | 07:04 |
| SpamapS | (this behavior is the same whether I use the Ubuntu-13.10 wheels mirror or not) | 07:04 |
| StevenK | SpamapS: Oh, and the 301 is from mod_speling, and if it's off, the response is 404 instead? | 07:05 |
| SpamapS | I'm also seeing the same fail Lotus907efi_ was seeing.. easy_install can't seem to get pbr right | 07:05 |
| SpamapS | 127.0.0.1 - - [19/Mar/2014:00:00:28 -0700] "GET /pypi/mirror/Ubuntu-13.10/pbr/ HTTP/1.1" 200 627 "-" "Python-urllib/2.7 setuptools/3.3" | 07:05 |
| SpamapS | StevenK: correct | 07:05 |
| SpamapS | StevenK: and there is no follow-up to prettytable | 07:05 |
| StevenK | loopback vfat filesystem? :-P | 07:05 |
| SpamapS | StevenK: don't think I haven't toyed with the idea. ;) | 07:06 |
| StevenK | Haha | 07:06 |
| StevenK | Apparently, you can trick samba into doing it as well | 07:06 |
| StevenK | I can recall the topic of a blog post that I read a while back | 07:06 |
| lifeless | SpamapS: is it on by default ? | 07:07 |
| StevenK | mod_speling is not, IIRC | 07:07 |
| SpamapS | lifeless: it is not | 07:07 |
| SpamapS | you have to a2enmod it | 07:07 |
| SpamapS | or load it elsewhere | 07:07 |
| SpamapS | lifeless: are you using --offline ? | 07:08 |
| *** ramishra has joined #tripleo | 07:15 | |
| lifeless | SpamapS: did you use pypi-mirror or bandersnatch to make your mirror ? | 07:15 |
| lifeless | SpamapS: because I'll be pypi-mirror | 07:15 |
| lifeless | and missing metadata | 07:15 |
| lifeless | or something | 07:16 |
| lifeless | hmm, nothing obvious to cause tat | 07:17 |
| *** rpodolyaka1 has quit IRC | 07:17 | |
| SpamapS | lifeless: pypi-mirror | 07:17 |
| lifeless | but hitting e.g. prettytable in my mirror is 404 not magic or redirects | 07:17 |
| SpamapS | lifeless: PrettyTable is 404 for me | 07:17 |
| SpamapS | lifeless: so it may be an ordering thing? | 07:17 |
| SpamapS | as in.. | 07:18 |
| SpamapS | global reqs may have changed from one to the other between our first runs | 07:18 |
| lifeless | SpamapS: do you mean, if we have PrettyTable somewhere | 07:19 |
| lifeless | once its installed a bad ref to prettytable is ignored? | 07:19 |
| lifeless | but this has worked for weeks for me | 07:19 |
| SpamapS | I made my pypi-mirror about a week ago | 07:19 |
| SpamapS | lifeless: on disk, do you have 'prettytable' or 'PrettyTable' ? | 07:20 |
| SpamapS | lifeless: I _only_ have 'prettytable' | 07:20 |
| lifeless | SpamapS: this is a bandersnatch mirror | 07:20 |
| lifeless | I only have PrettyTable | 07:20 |
| SpamapS | interesting | 07:21 |
| lifeless | which since its bandersnatch | 07:22 |
| lifeless | means thats the official name | 07:22 |
| SpamapS | what if it is as simple as pypi.python.org/simple changed? | 07:22 |
| lifeless | and pypi-mirror is mirroring incorrectly for some reason | 07:22 |
| SpamapS | lifeless: there me be an invalid assumption.. | 07:26 |
| *** cwolferh has joined #tripleo | 07:27 | |
| SpamapS | lifeless: https://pypi.python.org/pypi/PrettyTable <-- points at prettytable-* for download | 07:27 |
| lifeless | yes | 07:28 |
| lifeless | and | 07:28 |
| SpamapS | yep.. think I found it | 07:28 |
| *** ramishra has quit IRC | 07:29 | |
| openstackgerrit | Ladislav Smola proposed a change to openstack/tripleo-image-elements: Setting negative RamWeigher https://review.openstack.org/81200 | 07:29 |
| *** jtomasek has quit IRC | 07:30 | |
| SpamapS | lifeless: https://git.openstack.org/cgit/openstack-infra/pypi-mirror/tree/pypi_mirror/cmd/run_mirror.py#n375 | 07:30 |
| *** matsuhashi has quit IRC | 07:30 | |
| SpamapS | Should be that package_name is used to create the index dir | 07:30 |
| SpamapS | err | 07:30 |
| SpamapS | yeah | 07:30 |
| *** matsuhashi has joined #tripleo | 07:31 | |
| lifeless | isn't packagename being inferred there ? | 07:32 |
| lifeless | via a regex thus the issue ? | 07:32 |
| SpamapS | so it is using the regex on the _file_ as the package name | 07:33 |
| SpamapS | and then creating a dir for the package name | 07:33 |
| SpamapS | instead of just using the same parent dir | 07:33 |
| SpamapS | Not sure why pypi.openstack.org users don't see this | 07:35 |
| SpamapS | HEAD http://pypi.openstack.org/openstack/PrettyTable/ | 07:35 |
| SpamapS | User-Agent: lwp-request/6.03 libwww-perl/6.05 | 07:35 |
| SpamapS | 404 Not Found | 07:35 |
| *** xuhaiwei has joined #tripleo | 07:37 | |
| xuhaiwei | lifeless: Can we use one undercloud to create more than one overcloud? | 07:40 |
| *** morganfainberg is now known as morganfainberg_Z | 07:43 | |
| SpamapS | xuhaiwei: yes definitely | 07:45 |
| SpamapS | xuhaiwei: we have done that for our CI purposes :) | 07:46 |
| StevenK | Huh, some of the CI jobs have failed with PrettyTable | 07:46 |
| StevenK | And some are passing | 07:46 |
| xuhaiwei | can we do that(create more than one overcloud) using devtest? | 07:48 |
| SpamapS | xuhaiwei: no | 07:49 |
| SpamapS | xuhaiwei: but you can use the devtest created undercloud to manually drive Heat and create a second overcloud. | 07:49 |
| SpamapS | jang: https://review.openstack.org/81469 <-- the "spellcheck" problem solved. | 07:50 |
| *** rdopieralski has joined #tripleo | 07:52 | |
| *** e0ne has joined #tripleo | 07:52 | |
| *** ramishra has joined #tripleo | 07:52 | |
| StevenK | SpamapS: You can make that a little cleaner | 07:54 |
| StevenK | SpamapS: name_match is no longer referenced, so you can collapse it down to if PACKAGE_VERSION_RE.search(tarball) is None: | 07:54 |
| *** nati_ueno has joined #tripleo | 07:55 | |
| *** ifarkas has quit IRC | 07:58 | |
| *** rdopieralski has quit IRC | 07:58 | |
| *** noslzzp has joined #tripleo | 08:02 | |
| *** CaptTofu has joined #tripleo | 08:04 | |
| *** athomas has joined #tripleo | 08:04 | |
| *** shakamunyi has quit IRC | 08:07 | |
| *** jtomasek has joined #tripleo | 08:08 | |
| GheRivero | morning TripleO! | 08:09 |
| *** CaptTofu has quit IRC | 08:09 | |
| *** e0ne has quit IRC | 08:11 | |
| *** martyntaylor has joined #tripleo | 08:16 | |
| *** mrunge has joined #tripleo | 08:19 | |
| *** jcoufal has joined #tripleo | 08:22 | |
| rpodolyaka | morning all | 08:22 |
| *** rpodolyaka1 has joined #tripleo | 08:24 | |
| *** martyntaylor1 has joined #tripleo | 08:24 | |
| SpamapS | StevenK: by all means, make it cleaner. I find the code a bit.. obtuse.. :-P | 08:25 |
| *** uvirtbot has joined #tripleo | 08:25 | |
| *** martyntaylor has quit IRC | 08:26 | |
| SpamapS | StevenK: however, it is time that I slept | 08:26 |
| * SpamapS sleeps | 08:27 | |
| *** martyntaylor has joined #tripleo | 08:27 | |
| *** rpodolyaka1 has quit IRC | 08:28 | |
| *** martyntaylor1 has quit IRC | 08:30 | |
| *** shakamunyi has joined #tripleo | 08:33 | |
| *** sseago has joined #tripleo | 08:33 | |
| *** shakamunyi has quit IRC | 08:38 | |
| *** yamahata has quit IRC | 08:51 | |
| jang | Just tried the latest pypi-mirror from git, by the way: it really does create tarballs alongside wheels, so that's rather an improvement. Thanks all. | 08:51 |
| *** jcoufal has quit IRC | 08:54 | |
| *** andreaf has joined #tripleo | 08:54 | |
| *** vkozhukalov_ has quit IRC | 08:55 | |
| *** al has quit IRC | 08:55 | |
| *** al has joined #tripleo | 08:56 | |
| lifeless | GheRivero: https://review.openstack.org/#/c/81443/ | 08:57 |
| lifeless | GheRivero: (firedrill in topic) | 08:57 |
| *** hashar has joined #tripleo | 08:57 | |
| GheRivero | lifeless: looking | 08:57 |
| *** bauzas has joined #tripleo | 09:02 | |
| *** e0ne has joined #tripleo | 09:03 | |
| GheRivero | lifeless: is there a bug for the prettytable issue? | 09:07 |
| openstackgerrit | A change was merged to openstack/tripleo-incubator: Set default values for repo location and ref https://review.openstack.org/81443 | 09:08 |
| *** e0ne has quit IRC | 09:20 | |
| *** e0ne has joined #tripleo | 09:22 | |
| *** rpodolyaka1 has joined #tripleo | 09:24 | |
| *** jcoufal has joined #tripleo | 09:24 | |
| *** jcoufal has quit IRC | 09:25 | |
| *** jcoufal has joined #tripleo | 09:26 | |
| *** rpodolyaka1 has quit IRC | 09:28 | |
| *** jrist has joined #tripleo | 09:32 | |
| *** nati_ueno has quit IRC | 09:33 | |
| *** nati_ueno has joined #tripleo | 09:34 | |
| *** shakamunyi has joined #tripleo | 09:34 | |
| *** lucasagomes has joined #tripleo | 09:36 | |
| mordred | jang: yay! | 09:38 |
| mordred | jang: so perhaps we should makea new release? | 09:38 |
| *** jp_at_hp has joined #tripleo | 09:38 | |
| *** shakamunyi has quit IRC | 09:38 | |
| *** nati_ueno has quit IRC | 09:38 | |
| *** bauzas has quit IRC | 09:39 | |
| lifeless | GheRivero: I'm not sure. There should be. | 09:40 |
| lifeless | mordred: no, we have a crucial fix to do first. | 09:40 |
| *** martyntaylor has quit IRC | 09:40 | |
| *** martyntaylor1 has joined #tripleo | 09:41 | |
| lifeless | mordred: which will also fix the use of file:// for you :) | 09:41 |
| lifeless | mordred: I added you as reviewer on it | 09:41 |
| *** lucasagomes has quit IRC | 09:41 | |
| openstackgerrit | Matthew Macdonald-Wallace proposed a change to openstack/tripleo-image-elements: Install the check_mk Agent https://review.openstack.org/81485 | 09:44 |
| *** e0ne_ has joined #tripleo | 09:46 | |
| *** nati_ueno has joined #tripleo | 09:47 | |
| *** e0ne has quit IRC | 09:49 | |
| *** Matt2 has quit IRC | 09:55 | |
| *** xuhaiwei has quit IRC | 10:00 | |
| *** lynxman has quit IRC | 10:04 | |
| *** CaptTofu has joined #tripleo | 10:05 | |
| *** lynxman has joined #tripleo | 10:05 | |
| *** jcoufal has quit IRC | 10:07 | |
| *** CaptTofu has quit IRC | 10:09 | |
| *** e0ne_ has quit IRC | 10:10 | |
| *** bauzas has joined #tripleo | 10:13 | |
| *** e0ne has joined #tripleo | 10:17 | |
| *** ramishra has quit IRC | 10:21 | |
| *** ramishra has joined #tripleo | 10:22 | |
| *** ramishra has quit IRC | 10:22 | |
| *** rpodolyaka1 has joined #tripleo | 10:25 | |
| *** rpodolyaka1 has quit IRC | 10:29 | |
| *** giulivo has joined #tripleo | 10:31 | |
| *** jistr|mobile has joined #tripleo | 10:31 | |
| *** vkozhukalov_ has joined #tripleo | 10:33 | |
| *** shakamunyi has joined #tripleo | 10:35 | |
| *** shakamunyi has quit IRC | 10:39 | |
| openstackgerrit | Matthew Macdonald-Wallace proposed a change to openstack/tripleo-image-elements: Install the check_mk Agent https://review.openstack.org/81485 | 10:40 |
| *** ramishra has joined #tripleo | 10:42 | |
| *** derekh has joined #tripleo | 10:54 | |
| openstackgerrit | Michael Kerrin proposed a change to openstack/diskimage-builder: WIP - enable elements to be re-playable https://review.openstack.org/81494 | 11:00 |
| openstackgerrit | Michael Kerrin proposed a change to openstack/diskimage-builder: Add script to replay elements in an existing image https://review.openstack.org/80267 | 11:00 |
| openstackgerrit | Michael Kerrin proposed a change to openstack/diskimage-builder: Add utility to mount and unmount images https://review.openstack.org/76509 | 11:00 |
| openstackgerrit | Michael Kerrin proposed a change to openstack/tripleo-image-elements: WIP - enable elements to be replayable https://review.openstack.org/81495 | 11:03 |
| *** nati_ueno has quit IRC | 11:06 | |
| *** ramishra has quit IRC | 11:07 | |
| *** jcoufal has joined #tripleo | 11:13 | |
| *** ccrouch has joined #tripleo | 11:15 | |
| openstackgerrit | Derek Higgins proposed a change to openstack/tripleo-incubator: Better describe the compatibility mode being echo'd https://review.openstack.org/80296 | 11:20 |
| *** e0ne has quit IRC | 11:21 | |
| *** matsuhashi has quit IRC | 11:22 | |
| *** matsuhashi has joined #tripleo | 11:24 | |
| *** rpodolyaka1 has joined #tripleo | 11:25 | |
| *** lucasagomes has joined #tripleo | 11:29 | |
| *** rpodolyaka1 has quit IRC | 11:29 | |
| *** fandi has quit IRC | 11:30 | |
| *** Matt2 has joined #tripleo | 11:30 | |
| *** shakamunyi has joined #tripleo | 11:36 | |
| *** matsuhashi has quit IRC | 11:37 | |
| StevenK | derekh: Hm. I'd rather we focused on *removing* the compatability mode entirely | 11:39 |
| derekh | SpamapS: Fair enough, I was just improving a message I saw and had no idea what it meant | 11:39 |
| StevenK | derekh: I'm not sure if lifeless finished refreshing the test environment, but https://review.openstack.org/#/c/78467/ does so, and I have a local commit that polishes it off. | 11:39 |
| *** shakamunyi has quit IRC | 11:40 | |
| openstackgerrit | Steve Kowalik proposed a change to openstack/tripleo-incubator: Remove backwards compatability in setup-baremetal https://review.openstack.org/78467 | 11:40 |
| StevenK | If it passes CI, the testenv has been refreshed. I shall chase tomorrow. | 11:41 |
| derekh | StevenK: k | 11:43 |
| StevenK | Oh, blah, it's missing a change to write-tripleorc | 11:47 |
| lifeless | derekh: I build a new testenv image and loaded it into glance in the hp region | 11:48 |
| lifeless | derekh: but CI was busy so I waited, then timed out to EOD :) | 11:48 |
| derekh | lifeless: ok, I don't suppose it included https://review.openstack.org/#/c/76171/3 | 11:50 |
| *** CaptTofu has joined #tripleo | 11:50 | |
| openstackgerrit | Steve Kowalik proposed a change to openstack/tripleo-incubator: Remove backwards compatability in setup-baremetal https://review.openstack.org/78467 | 11:51 |
| *** giulivo has quit IRC | 11:51 | |
| derekh | lifeless: or https://review.openstack.org/#/c/79663/ (this one will be needed) | 11:51 |
| *** e0ne has joined #tripleo | 11:52 | |
| derekh | any cores out there could look at https://review.openstack.org/#/c/79663/ and https://review.openstack.org/#/c/76171/3 please | 11:52 |
| Ng | derekh: sure | 11:52 |
| derekh | and https://review.openstack.org/#/c/80603/ | 11:52 |
| * Ng between talks atm | 11:52 | |
| derekh | Ng: thanks | 11:52 |
| Ng | this is not the biggest conference in the world ever :) | 11:52 |
| *** lucasagomes has quit IRC | 11:53 | |
| *** jrist has quit IRC | 11:53 | |
| derekh | Ng: just a tad smaller then the openstack summit? | 11:53 |
| Ng | derekh: at least one order of magnitude smaller :) | 11:53 |
| mordred | lifeless: awesome. thanks! | 11:54 |
| *** e0ne has quit IRC | 11:56 | |
| Ng | derekh: done :) | 11:56 |
| mordred | lifeless, SpamapS, Ng: SpamapS patch to pypi-mirror seems to be unhappy on integration gate | 11:58 |
| derekh | Ng: thanks | 12:02 |
| *** al has quit IRC | 12:02 | |
| *** al has joined #tripleo | 12:03 | |
| *** hashar has quit IRC | 12:04 | |
| *** e0ne has joined #tripleo | 12:07 | |
| *** e0ne_ has joined #tripleo | 12:10 | |
| *** e0ne has quit IRC | 12:10 | |
| openstackgerrit | A change was merged to openstack-infra/tripleo-ci: Allow the testenv client to dictate the worker timeout https://review.openstack.org/76171 | 12:11 |
| *** dprince has joined #tripleo | 12:11 | |
| *** bauzas has quit IRC | 12:12 | |
| openstackgerrit | A change was merged to openstack-infra/tripleo-ci: Install jq on Test Environment hosts https://review.openstack.org/79663 | 12:12 |
| *** hashar has joined #tripleo | 12:14 | |
| derekh | and another needed before building new test env hosts https://review.openstack.org/#/c/80265/ | 12:15 |
| derekh | and a nice to have https://review.openstack.org/#/c/80603/1 | 12:16 |
| *** bauzas has joined #tripleo | 12:17 | |
| *** jcoufal has quit IRC | 12:19 | |
| *** al has quit IRC | 12:24 | |
| *** al has joined #tripleo | 12:24 | |
| *** rpodolyaka1 has joined #tripleo | 12:26 | |
| *** e0ne has joined #tripleo | 12:28 | |
| *** rpodolyaka1 has quit IRC | 12:30 | |
| *** e0ne_ has quit IRC | 12:32 | |
| rcarrillocruz | guys, i'm getting 'libvirt.libvirtError: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Permission denied' when 'configure-vm' is called from 'setup-seed-vm' | 12:35 |
| rcarrillocruz | shouldn't it be called with sudo? | 12:35 |
| *** shakamunyi has joined #tripleo | 12:36 | |
| *** Killer_prince has quit IRC | 12:37 | |
| *** jtomasek has quit IRC | 12:37 | |
| *** weshay has joined #tripleo | 12:40 | |
| *** shakamunyi has quit IRC | 12:41 | |
| openstackgerrit | Michael Kerrin proposed a change to openstack/tripleo-image-elements: WIP - enable elements to be replayable https://review.openstack.org/81495 | 12:43 |
| openstackgerrit | Gonéri Le Bouder proposed a change to openstack/tripleo-incubator: Rest syntax fixes https://review.openstack.org/81012 | 12:43 |
| *** bauzas has quit IRC | 12:44 | |
| *** bauzas has joined #tripleo | 12:44 | |
| *** mrunge has quit IRC | 12:51 | |
| *** jrist has joined #tripleo | 12:58 | |
| openstackgerrit | Michael Kerrin proposed a change to openstack/tripleo-incubator: Cause wait_for heat stack-create fail fast https://review.openstack.org/76537 | 12:59 |
| *** lblanchard has joined #tripleo | 13:00 | |
| openstackgerrit | Jon-Paul Sullivan proposed a change to openstack/tripleo-image-elements: Create pip manifest files of installed packages https://review.openstack.org/80644 | 13:09 |
| openstackgerrit | Jon-Paul Sullivan proposed a change to openstack/tripleo-image-elements: Make os-refresh-config install.d hook run later https://review.openstack.org/81174 | 13:09 |
| *** hashar has quit IRC | 13:12 | |
| *** julim has joined #tripleo | 13:12 | |
| *** hashar has joined #tripleo | 13:14 | |
| *** nosnos has quit IRC | 13:24 | |
| *** rpodolyaka1 has joined #tripleo | 13:26 | |
| *** jrist has quit IRC | 13:27 | |
| *** rpodolyaka1 has quit IRC | 13:31 | |
| *** jrist has joined #tripleo | 13:36 | |
| *** shakamunyi has joined #tripleo | 13:37 | |
| *** funzo has quit IRC | 13:40 | |
| *** shakamunyi has quit IRC | 13:41 | |
| *** lucasagomes_ has joined #tripleo | 13:45 | |
| *** funzo has joined #tripleo | 13:50 | |
| *** nati_ueno has joined #tripleo | 13:50 | |
| *** jpeeler has quit IRC | 13:55 | |
| *** jpeeler has joined #tripleo | 13:56 | |
| *** nati_ueno has quit IRC | 14:03 | |
| *** jrist has quit IRC | 14:05 | |
| *** vkozhukalov_ has quit IRC | 14:05 | |
| *** lucasagomes_ has quit IRC | 14:06 | |
| *** vkozhukalov_ has joined #tripleo | 14:08 | |
| *** shakamunyi has joined #tripleo | 14:18 | |
| openstackgerrit | Alexis Lee proposed a change to openstack/tripleo-incubator: Use known dir instead of relying on BASH_SOURCE https://review.openstack.org/81539 | 14:22 |
| *** rpodolyaka1 has joined #tripleo | 14:27 | |
| bnemec | rcarrillocruz: You may need to log out and log back in. | 14:31 |
| *** rpodolyaka1 has quit IRC | 14:31 | |
| bnemec | rcarrillocruz: See https://bugs.launchpad.net/tripleo/+bug/1285285 | 14:31 |
| uvirtbot | Launchpad bug 1285285 in tripleo "libvirtd group addition doesn't stick" [Medium,Fix released] | 14:32 |
| bnemec | (note that the bug status lies - the fix was reverted because CI somehow depends on this broken behavior) | 14:32 |
| *** shakamunyi has quit IRC | 14:52 | |
| *** shakamunyi has joined #tripleo | 14:52 | |
| rcarrillocruz | guys, is ubuntu 12.04 supported in tripleo? i found some mixed opiniones on this in email threads and such | 14:53 |
| rcarrillocruz | https://bugs.launchpad.net/tripleo/+bug/1292938 | 14:53 |
| uvirtbot | Launchpad bug 1292938 in tripleo "dkms fails due to devtest not requiring linux-headers package as dependency" [Undecided,Triaged] | 14:53 |
| rcarrillocruz | i can't repro on 13.10, but i can on 12.04 | 14:53 |
| rcarrillocruz | if we don't support it (or we are about to ditch support), then it's not worth working on a fix | 14:53 |
| rcarrillocruz | thanks bnemec, i just made it work randomly by closing/reopening my session after lunch, and now i just read your comment, hehe | 14:54 |
| SpamapS | mordred: ack | 14:55 |
| *** jrist has joined #tripleo | 14:59 | |
| *** morazi has joined #tripleo | 15:01 | |
| openstackgerrit | Allison Randal proposed a change to openstack/tripleo-heat-templates: Pass undercloud metadata to overcloud compute https://review.openstack.org/81552 | 15:07 |
| *** yamahata has joined #tripleo | 15:07 | |
| jp_at_hp | tchaypo: did you see that I updated https://review.openstack.org/#/c/81174/ to rebase it on slagles change? I'm hoping that was OK? I just wanted to get my updated patch up, as I was asked to rebase it onto your change. | 15:14 |
| *** spzala has joined #tripleo | 15:15 | |
| SpamapS | jp_at_hp: It is highly encouraged to "do the right thing" :) | 15:19 |
| SpamapS | jp_at_hp: A few months back we sent out some messages on the openstack-dev list that encouraged everyone to say "Thanks for fixing my patch." :) | 15:20 |
| openstackgerrit | Coleman Corrigan proposed a change to openstack/tripleo-incubator: use DIB_REPOLOCATION_ if set in pull-tools script https://review.openstack.org/81554 | 15:20 |
| *** CaptTofu has quit IRC | 15:21 | |
| *** CaptTofu has joined #tripleo | 15:22 | |
| jp_at_hp | SpamapS: it's just the unintended side-effects of bug reassignment, etc... | 15:23 |
| *** CaptTofu has quit IRC | 15:26 | |
| *** rpodolyaka1 has joined #tripleo | 15:28 | |
| *** CaptTofu has joined #tripleo | 15:28 | |
| *** e0ne has quit IRC | 15:29 | |
| SpamapS | jp_at_hp: Right, the machinery isn't perfectly oriented, but the community embraces it nonetheless. | 15:29 |
| *** rpodolyaka1 has quit IRC | 15:32 | |
| *** rpodolyaka has quit IRC | 15:35 | |
| *** rpodolyaka has joined #tripleo | 15:36 | |
| *** bauzas1 has joined #tripleo | 15:38 | |
| *** bauzas has quit IRC | 15:38 | |
| *** jcoufal has joined #tripleo | 15:39 | |
| *** jcoufal has quit IRC | 15:39 | |
| *** petertoft has joined #tripleo | 15:42 | |
| *** bauzas1 has quit IRC | 15:42 | |
| *** bauzas has joined #tripleo | 15:42 | |
| *** newell has joined #tripleo | 15:45 | |
| *** lucasagomes_ has joined #tripleo | 15:48 | |
| *** e0ne has joined #tripleo | 15:53 | |
| andrearosa | git status | 15:55 |
| *** ccrouch has quit IRC | 16:02 | |
| openstackgerrit | Allison Randal proposed a change to openstack/tripleo-incubator: Pass undercloud metadata to overcloud compute https://review.openstack.org/81572 | 16:10 |
| *** rlandy has quit IRC | 16:11 | |
| derekh | can I get some eyes on https://review.openstack.org/#/c/80265/ please , | 16:12 |
| derekh | we'll need that ^^ before creating a new TE image (https://review.openstack.org/#/c/80603/ would be nice too, but not needed) | 16:13 |
| derekh | lifeless: FYI, once that merges we can build/deploy an image | 16:14 |
| *** noslzzp has quit IRC | 16:20 | |
| *** athomas has quit IRC | 16:21 | |
| bnemec | slagle: derekh: https://review.openstack.org/#/c/74867/9 passed devtest against Fedora for me locally and it passed CI yesterday. | 16:21 |
| *** lucasagomes_ has quit IRC | 16:21 | |
| bnemec | The dependent patch failed on a heisenbug, but since none of this stuff is actually tested in CI the test results there are kind of meaningless. | 16:22 |
| *** dprince has quit IRC | 16:23 | |
| derekh | bnemec: A+'d the one thats passed, agreed non of this is tested in the ci (and passed for me yesterday), sorry gotta wait on ci because its now a formality we need to follow, but will A+ once its done | 16:25 |
| *** jrist has quit IRC | 16:25 | |
| *** morazi has quit IRC | 16:25 | |
| bnemec | derekh: No problem, that's how it should be. Thanks for looking. | 16:25 |
| derekh | bnemec: no prob | 16:26 |
| jp_at_hp | derekh: On https://review.openstack.org/#/c/80265/ - the commit message has about 3 grammar/spelling mistakes in it... | 16:26 |
| derekh | jp_at_hp: sounds about right, fixing, thanks | 16:26 |
| jp_at_hp | sorry - when it's just a commit emssage and not code I feel bad bringing it up :( | 16:27 |
| jp_at_hp | because I obviously can't type... | 16:27 |
| bnemec | Muphry's law strikes again! | 16:28 |
| openstackgerrit | Derek Higgins proposed a change to openstack/tripleo-incubator: Ensure json file is initialised https://review.openstack.org/80265 | 16:28 |
| *** rpodolyaka1 has joined #tripleo | 16:28 | |
| *** bauzas has quit IRC | 16:28 | |
| derekh | jp_at_hp: no prob, hope that's better. | 16:30 |
| *** martyntaylor1 has quit IRC | 16:32 | |
| *** rpodolyaka1 has quit IRC | 16:32 | |
| *** UtahDave has joined #tripleo | 16:34 | |
| *** UtahDave has left #tripleo | 16:36 | |
| NobodyCam | question on a odd seed vm error.. anyone encountering a strange error with rabbit saying it started but nothing listening on port 5672? | 16:38 |
| *** hashar has quit IRC | 16:39 | |
| *** shakamunyi has quit IRC | 16:40 | |
| *** jistr|mobile has quit IRC | 16:42 | |
| *** sseago has quit IRC | 16:46 | |
| *** dprince has joined #tripleo | 16:47 | |
| *** sseago has joined #tripleo | 16:48 | |
| greghaynes | I havent recently but there were 3 or so rabbit patches that merged yesterday | 16:49 |
| clarkb | SpamapS: pip shouldn't be case sensitive | 16:49 |
| clarkb | SpamapS: but if it is you can always use apache | 16:50 |
| NobodyCam | greghaynes: ahhh | 16:50 |
| *** epim has joined #tripleo | 16:51 | |
| NobodyCam | greghaynes: have you built a seed sense they landed? | 16:52 |
| greghaynes | Nope, working on that now... ive run into a different bug while trying to build after recent merges | 16:52 |
| greghaynes | so hopefully soon | 16:52 |
| NobodyCam | :) greghaynes TY | 16:52 |
| bnemec | I just built a seed on Fedora this morning that I believe had the latest rabbitmq changes. | 16:53 |
| bnemec | Doesn't mean it's not broken on Ubuntu though. | 16:53 |
| NobodyCam | bnemec: ack | 16:56 |
| greghaynes | huh, now I cant just run pull-tools | 16:58 |
| bnemec | greghaynes: Do you have https://review.openstack.org/#/c/81443/ ? | 16:58 |
| greghaynes | Good call | 16:58 |
| *** akuznetsov has quit IRC | 16:58 | |
| *** andreaf has quit IRC | 16:59 | |
| NobodyCam | greghaynes: bnemec: this is what I am seeing http://paste.openstack.org/show/YFnkjtyojkHUp2zDZqMF/ | 17:00 |
| *** markmc has quit IRC | 17:03 | |
| greghaynes | Whats the problem thre? | 17:03 |
| NobodyCam | greghaynes: note the netstat nothing on 5672 | 17:03 |
| greghaynes | .. huh | 17:05 |
| greghaynes | wierd | 17:05 |
| NobodyCam | ya | 17:07 |
| bnemec | For comparison, here's what my working seed looks like: http://paste.openstack.org/show/73837/ | 17:07 |
| greghaynes | my devtest is spinning, shouldnt be much longer | 17:07 |
| bnemec | Not sure what the difference between beam and epmd | 17:07 |
| greghaynes | just enough time for coffee \O/ | 17:07 |
| *** derekh has quit IRC | 17:07 | |
| NobodyCam | bnemec: ya line 26 shows rabbit listening | 17:08 |
| bnemec | Hmm, mine is only listening on ipv6 5672. | 17:08 |
| greghaynes | I think that also means ipv4 | 17:08 |
| NobodyCam | ya | 17:08 |
| *** e0ne has quit IRC | 17:09 | |
| NobodyCam | brb | 17:09 |
| *** vkozhukalov_ has quit IRC | 17:14 | |
| bnemec | rwsu: Is rabbitmq still broken with selinux? | 17:14 |
| bnemec | Wondering if I can just fire off a devtest run with your mysql patch and expect it to work. | 17:14 |
| greghaynes | NobodyCam: My seed seems to of built fine on ubuntu | 17:15 |
| *** morganfainberg_Z is now known as morganfainberg | 17:15 | |
| NobodyCam | greghaynes: :) humm | 17:16 |
| rwsu | bnemec: yup still broken, runs veerry slow | 17:16 |
| rwsu | bnemec: you'll run into other issues with selinux turned on even with mysql patch | 17:17 |
| bnemec | rwsu: Okay, thanks. I won't try that then. :-) | 17:17 |
| *** Lotus907efi_ has quit IRC | 17:17 | |
| *** ccrouch has joined #tripleo | 17:20 | |
| *** noslzzp has joined #tripleo | 17:29 | |
| *** rpodolyaka1 has joined #tripleo | 17:29 | |
| *** rpodolyaka1 has quit IRC | 17:33 | |
| jp_at_hp | slagle: SpamapS on https://review.openstack.org/#/c/81174/ - should the package version of the install also move from 01 to 10 to maintain ordering in uses of the element that use different install types? | 17:40 |
| *** weshay has quit IRC | 17:42 | |
| *** weshay has joined #tripleo | 17:42 | |
| *** sseago has quit IRC | 17:42 | |
| *** weshay has quit IRC | 17:43 | |
| *** weshay has joined #tripleo | 17:43 | |
| *** akuznetsov has joined #tripleo | 17:44 | |
| *** cwolferh_ has joined #tripleo | 18:02 | |
| *** cwolferh has quit IRC | 18:04 | |
| SpamapS | jp_at_hp: well.. probably for consistency yes | 18:06 |
| SpamapS | jp_at_hp: but the move isn't necessary in that particular case because packages don't need pip ;) | 18:06 |
| jp_at_hp | SpamapS: not really my point. Having them inconsistent could possibly introduce bugs seen only when using packages or only when using source, which would be a bad thing. | 18:07 |
| *** epim has quit IRC | 18:09 | |
| *** petertoft has quit IRC | 18:09 | |
| NobodyCam | greghaynes: bnemec: this is a odd error: http://paste.openstack.org/show/T5dBmP0arbHOnC2iZgIn/ | 18:09 |
| *** rpodolyaka1 has joined #tripleo | 18:09 | |
| SpamapS | jp_at_hp: Yes it could possibly be confusing I agree. However, using packages is more likely to introduce bugs that only happen when using packages.. so I see it as a low priority. ;) Good catch though! | 18:10 |
| greghaynes | NobodyCam: Interesting, check out your heat template | 18:10 |
| greghaynes | oh, this is for your seed? | 18:11 |
| jp_at_hp | SpamapS: I would place it as more important than grammar in a README though. | 18:11 |
| NobodyCam | greghaynes: yep seed | 18:12 |
| greghaynes | NobodyCam: just for sanitys sake - have you rebased the patch your working on in t-i off master recently? | 18:14 |
| greghaynes | Not sure why the rabbit changes that went in to t-i-e would want that but *shrug* | 18:15 |
| *** jtomasek has joined #tripleo | 18:15 | |
| bnemec | NobodyCam: Could you have an older seed config.json? | 18:15 |
| bnemec | https://github.com/openstack/tripleo-image-elements/blob/master/elements/rabbitmq-server/os-config-applier/mnt/state/var/lib/rabbitmq/.erlang.cookie#L1 | 18:15 |
| bnemec | I assume that needs to be set in it for this to work. | 18:16 |
| bnemec | I bet you're missing https://github.com/openstack/tripleo-image-elements/blob/master/elements/seed-stack-config/config.json#L16 | 18:18 |
| NobodyCam | bnemec: I'll bet your correct TY testing now | 18:20 |
| slagle | jp_at_hp: yea, i've kept the numbering the same, just for consistency really | 18:31 |
| jp_at_hp | thanks slagle | 18:32 |
| slagle | the other aspect of it is if there are common install scripts that are just under install.d, you might want that to always run before (or after) the implementation under *-{package,source}-install | 18:32 |
| jp_at_hp | Now if only there were some cores to review ;-) | 18:32 |
| SpamapS | jp_at_hp: why do we have to assign a value to either thing? They should both be corrected. | 18:33 |
| jp_at_hp | good point slagle - that's similar to what I was thinking of in other elements being dependent upon os-refresh-config. That's definitely a more up=close and personalexample of why keeping them the same would be good. | 18:33 |
| *** andreaf has joined #tripleo | 18:33 | |
| jp_at_hp | SpamapS: and lo - they are :D | 18:34 |
| *** sseago has joined #tripleo | 18:37 | |
| openstackgerrit | Dan Prince proposed a change to openstack/diskimage-builder: DHCP: make udev rules want dhcp-interface@.service https://review.openstack.org/81596 | 18:42 |
| NobodyCam | bnemec: greghaynes: TY seems to have gotten me past that issue... :) | 18:42 |
| bnemec | \o/ | 18:43 |
| *** spzala has quit IRC | 18:44 | |
| greghaynes | w00t, that means we get ironic support now, right? | 18:44 |
| greghaynes | ;) | 18:44 |
| NobodyCam | :) we'll see attempting to deployundercloud via ironic. | 18:45 |
| NobodyCam | right now | 18:45 |
| *** akuznetsov has quit IRC | 18:45 | |
| *** shakayumi has joined #tripleo | 18:47 | |
| *** shakayumi has quit IRC | 18:47 | |
| *** julim has quit IRC | 18:47 | |
| *** saurabhs has joined #tripleo | 18:57 | |
| *** jp_at_hp has quit IRC | 18:58 | |
| *** shakayumi has joined #tripleo | 18:58 | |
| *** noslzzp has quit IRC | 19:01 | |
| *** shakayumi has joined #tripleo | 19:02 | |
| *** derekh has joined #tripleo | 19:02 | |
| *** noslzzp has joined #tripleo | 19:03 | |
| saurabhs | lifeless: for trove disk image builder elements, I am getting a warning 'WARNING: first-boot.d is DEPRECATED, it will be removed rom diskimage-builder in a future release' can you tell me what should be a replacement for first-boot.d going forward | 19:03 |
| lifeless | saurabhs: idempotent os-refresh-config scripts | 19:03 |
| *** shakayumi has joined #tripleo | 19:04 | |
| derekh | looks like our geard fell over again, restarted now currently running ci jobs will probably fail, but should be ok after that | 19:04 |
| *** e0ne has joined #tripleo | 19:04 | |
| clarkb | its interesting that your geard falls over a lot | 19:04 |
| clarkb | the logstash geard and zuul geard seem mostly solid | 19:04 |
| NobodyCam | undercloud | CREATE_IN_PROGRESS & undercloud-undercloud-vtdzoh2xs7ai | BUILD | spawning lloking good so far! | 19:05 |
| derekh | clarkb: its because we weren't closing connections to it, we have to build new test environments with the fix and deploy them | 19:06 |
| clarkb | oh right you go through objects | 19:06 |
| saurabhs | lifeless: os-refresh-config runs everytime there is metatata change. firstboot.d runs only when machine boots up for the first time. are they both different? or is there something in os-refresh-config that runs only on first boot | 19:07 |
| derekh | clarkb: yup, and eventually hit error: [Errno 24] Too many open files | 19:07 |
| derekh | lifeless: before we can deploy new testenv's we need https://review.openstack.org/#/c/80265/ (and ideally a new release of geard the new release is waiting on https://review.openstack.org/#/c/80304/) | 19:09 |
| SpamapS | saurabhs: if [ -e /some/state/file ] ; then exit 0 ; fi ... do idempotent first boot thing here ... ; touch /some/state/file | 19:10 |
| lifeless | derekh: i | 19:11 |
| lifeless | derekh: huh, where are we passing an absent file in ? | 19:14 |
| derekh | lifeless: https://github.com/openstack-infra/tripleo-ci/blob/master/elements/testenv-worker/bin/ensure-test-env#L33 | 19:15 |
| lifeless | derekh: oh, I see. So I want us to actually call into devtest_testenv.sh there. Hmm, I guess the handle missing path is ok | 19:16 |
| derekh | lifeless: yup, agreed, I intended to do that but keep getting pulled onto fixing what we currently have | 19:17 |
| NobodyCam | greghaynes: bnemec: undercloud deployed from seed via Ironic using USE_IRONIC=1; devtest.sh ... patches to follow after lunch | 19:17 |
| greghaynes | \O/ | 19:17 |
| lifeless | derekh: ok, so land that, because we don't CI create-nodes, it can't possibly affect CI one way or another | 19:18 |
| NobodyCam | :) | 19:18 |
| greghaynes | patches or it didnt happen :p | 19:18 |
| lifeless | SpamapS: https://review.openstack.org/#/c/80265/ ^ | 19:18 |
| *** sseago has quit IRC | 19:18 | |
| NobodyCam | greghaynes: lunch first ! | 19:19 |
| NobodyCam | :-p | 19:19 |
| *** shakayumi has quit IRC | 19:19 | |
| SpamapS | derekh: https://review.openstack.org/#/c/80265/ -1'd for commit message .. fix quick pls will review again now | 19:19 |
| NobodyCam | greghaynes: http://paste.openstack.org/show/vykcN27wlPuhrrVV3aov/ | 19:20 |
| greghaynes | noice | 19:20 |
| NobodyCam | :) | 19:20 |
| openstackgerrit | Derek Higgins proposed a change to openstack/tripleo-incubator: Ensure json file is initialised https://review.openstack.org/80265 | 19:21 |
| lifeless | devananda: hi | 19:21 |
| lifeless | actually nvm | 19:21 |
| derekh | SpamapS: thanks, done | 19:21 |
| *** rpodolyaka1 has quit IRC | 19:22 | |
| *** spzala has joined #tripleo | 19:23 | |
| *** vkozhukalov_ has joined #tripleo | 19:23 | |
| *** rpodolyaka1 has joined #tripleo | 19:25 | |
| *** spzala has quit IRC | 19:26 | |
| derekh | lifeless: so I've been running a script that I think gives a nice overview of how our jobs are doing http://goodsquishy.com/downloads/tripleo-jobs.html , I've found it very handy to monitor in order to quickly identify problems, needs a more permanent home though | 19:27 |
| lifeless | derekh: elastic-rechecj ? :) | 19:28 |
| *** Lotus907efi has joined #tripleo | 19:28 | |
| *** petertoft has joined #tripleo | 19:29 | |
| derekh | lifeless: ya, I've been toying with elasic recheck aswell, but will it tell us things like "none of our overcloud jobs has passed since 2pm yesterday" , these things are going unnoticed until people spend time looking into why ci failed on their last 3 attempts by which time a day has passed | 19:30 |
| derekh | lifeless: would elastic recheck only help if we add queries for known intermittent failures | 19:31 |
| lifeless | jogo: ^ | 19:31 |
| derekh | lifeless: Of which we currently have one http://logstash.openstack.org/#eyJzZWFyY2giOiJAbWVzc2FnZVwiQ2Fubm90IGZldGNoIGluZGV4IGJhc2UgVVJMXCIgQU5EIGJ1aWxkX25hbWU6Y2hlY2stdHJpcGxlby0qLXByZWNpc2UiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjYwNDgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjEzOTUxNjUxMzY0NjksIm1vZGUiOiIiLCJhbmFseXplX2ZpZWxkIjoiIn0= | 19:31 |
| derekh | lifeless: I can add that queries to elastic recheck for tripleo jobs which will help in that instance | 19:32 |
| NobodyCam | TripleO element question, would it be better to rename and create new elements for seed-stack-config.. ie. seed-stack-config-nova & seed-stack-config-ironic.. or keep single seed-stack-config element? | 19:32 |
| lifeless | NobodyCam: single one probably | 19:34 |
| NobodyCam | lifeless: ack :) | 19:35 |
| greghaynes | lifeless: So I think the galera 'works' now with 5.5, I must of been trying to debug 5.6 without realizing when it was OOMing | 19:36 |
| greghaynes | \O/ galera might actually merge now | 19:36 |
| greghaynes | also, all tests seemed to of passed | 19:36 |
| SpamapS | greghaynes: s/must of/must have/ (and yes, I will correct it every time I see it) | 19:38 |
| SpamapS | because I am working on transitioning to full grumpy old man status | 19:38 |
| greghaynes | argh, and I thought I was getting better | 19:38 |
| *** julim has joined #tripleo | 19:38 | |
| greghaynes | also, seem to have passed | 19:39 |
| SpamapS | Converting Heat templates to softwareconfig is helping quite a bit with the "get grumpy" part | 19:39 |
| greghaynes | haha | 19:39 |
| *** fandi has joined #tripleo | 19:41 | |
| *** petertoft has quit IRC | 19:43 | |
| openstackgerrit | A change was merged to openstack/tripleo-incubator: Ensure json file is initialised https://review.openstack.org/80265 | 19:44 |
| *** hashar has joined #tripleo | 19:47 | |
| derekh | lifeless: submitted recheck query for that pip bug https://review.openstack.org/#/c/81609/ | 19:48 |
| lifeless | greghaynes: wicked | 19:52 |
| derekh | btw, looks like the root of that intermittent pip error seems to be caused by pip not receiving a response from the server before a 15 second timeout (digging into pip code now to figure out if there is a way around it) | 19:57 |
| lifeless | derekh: yes, this is why I want local mirrors of stuff | 19:58 |
| lifeless | derekh: we have regular network blips, and we don't know why | 19:58 |
| derekh | we only see the problem in packages where cases mismatch because pip tries to get the URL 2 ways, when there is a case mismatch we have no failabck and are mor likely to fail | 19:59 |
| Lotus907efi | I have another person here who has a SL390 with 24 cores and 94GB of memory and she is trying to run devtest.sh using the pypi internal mirror I have setup as well as the internal Ubuntu saucy repo | 20:00 |
| Lotus907efi | but her devtest.sh run is taking 6+ hours | 20:00 |
| Lotus907efi | which is way too long | 20:01 |
| derekh | in the case I managed to catch with tcpdump, it took 21 seconds before we got a response from server | 20:01 |
| Lotus907efi | I am looking at her output file and the thing looks like it is using the mirrors | 20:01 |
| derekh | Lotus907efi: the only time I saw really long times like that kvm wasn't loaded, sudo lsmod | grep -i kvm (allthough if your time is spent building images then thats not the problem) | 20:03 |
| Lotus907efi | hmmm | 20:04 |
| lifeless | Lotus907efi: where is the time going - image builds or vm deploys ? | 20:06 |
| *** epim has joined #tripleo | 20:06 | |
| Lotus907efi | I am trying to examine the output file I have to see what the heck is happening | 20:06 |
| *** CaptTofu has quit IRC | 20:06 | |
| lifeless | Lotus907efi: can you pastebin it? or something? | 20:07 |
| Lotus907efi | ummm, it is 2.8MB | 20:07 |
| Lotus907efi | let me see if I can figure out a way to do that | 20:08 |
| Lotus907efi | I could attach it as a file to a bug :) | 20:08 |
| lifeless | you can pastebin 2.8MB | 20:08 |
| lifeless | pastebinit < file.name | 20:09 |
| Lotus907efi | a bug for "devtest takes WAY TOO LONG!" :) | 20:09 |
| Lotus907efi | ah, ok | 20:09 |
| openstackgerrit | James Slagle proposed a change to openstack/tripleo-image-elements: Always create o-a-c's $TEMPLATE_ROOT https://review.openstack.org/81614 | 20:09 |
| SpamapS | There's a little program on Ubuntu.. | 20:09 |
| SpamapS | kvm-ok | 20:09 |
| lifeless | derekh: ok so do you want me to deploy a new testenv, or will you ? | 20:12 |
| openstackgerrit | Ben Nemec proposed a change to openstack/tripleo-image-elements: Add cinder-lio element https://review.openstack.org/78463 | 20:13 |
| openstackgerrit | Ben Nemec proposed a change to openstack/tripleo-image-elements: Factor out tgt-specific parts of cinder element https://review.openstack.org/78462 | 20:13 |
| derekh | lifeless: I'll kick off a image build now, on train at the moment so not sure how far I'll get, will let you know | 20:14 |
| lifeless | derekh: ack ack | 20:14 |
| Lotus907efi | hmm, I tried to pastebin the log and it said: Bad API request, maximum paste file size exceeded | 20:16 |
| lifeless | Lotus907efi: hah nuts :) | 20:16 |
| lifeless | Lotus907efi: upload it to swift? | 20:16 |
| lifeless | I hear we have this cloud thing | 20:17 |
| Lotus907efi | hmmm, I don't have access to any swift datastores | 20:17 |
| Lotus907efi | so submit a dummy bug and then attach the log to that? | 20:18 |
| Lotus907efi | lifeless: are you or greg on hipchat? | 20:19 |
| lifeless | often | 20:19 |
| lifeless | Lotus907efi: I'm in the TripleO room | 20:20 |
| *** rpodolyaka1 has quit IRC | 20:20 | |
| openstackgerrit | Dan Prince proposed a change to openstack/diskimage-builder: serial-console: Use udev rules to startup getty https://review.openstack.org/81616 | 20:23 |
| *** rpodolyaka1 has joined #tripleo | 20:24 | |
| tchaypo | Lotus907efi: my first run on a desktop -grade machine took longer than that, but since I set up squid and the pypi-mirror element (the one that caches things in ~/.cache/image-create/pypi, not a real pypi mirror) my times have dropped right down | 20:24 |
| Lotus907efi | hmmm | 20:24 |
| Lotus907efi | well I would hardly call a 24 core system with 94GB of memory a desktop grade system :) | 20:24 |
| Lotus907efi | or at least I wouldn't mide having it as a desktop system | 20:25 |
| Lotus907efi | s/mide/mind | 20:25 |
| *** rpodolyaka1 has quit IRC | 20:28 | |
| *** Meena has joined #tripleo | 20:29 | |
| tchaypo | yeah, I'd expect it to be able to cope better | 20:30 |
| tchaypo | although, from what I've seen, my system doesn't seem to be CPU-bound | 20:31 |
| jogo | derekh: that is correct | 20:31 |
| tchaypo | the first few runs it was network-bound; once it had most things cached it seems to spend most of its time building disk images | 20:31 |
| *** CaptTofu has joined #tripleo | 20:32 | |
| jogo | derekh: so the basic log workflow that gate uses is: | 20:33 |
| jogo | devstack-gate runs tempest against cloud and we collect logs | 20:33 |
| jogo | logs (tempest,openstack, etc) are stored in file server and a subset are pushed into a massive elsticsearch cluster | 20:33 |
| jogo | elastic-recheck is a collection of elasticsearch fingerprints for bugs | 20:34 |
| jogo | ex: if you see this stacktrace then you hit bug x. | 20:34 |
| jogo | elastic-recheck is used for two things: | 20:34 |
| jogo | listen to gerrit stream and help developers figure out if there job failed due to a known bug | 20:35 |
| jogo | and secondly help us track race conditions and bugs | 20:35 |
| jogo | via http://status.openstack.org/elastic-recheck/ and http://status.openstack.org/elastic-recheck/data/uncategorized.html | 20:35 |
| derekh | jogo: yup and in order for us to get feedback from elastic recheck we need to configure it with known fingerprint e.g. https://review.openstack.org/#/c/81609/ | 20:36 |
| lifeless | derekh: so, I like your script, but think it should build on es :) | 20:36 |
| jogo | well thats assuming you have all the data in a elasticsearch cluster | 20:36 |
| derekh | lifeless so basically we need something to alert us to the fact that ci is failing all around us and its not obvious until somebody sees enough failures to dig in a take a look at why, e.g. the nova issue that last hit us was a problem for 4 days (although 2 were weekend) | 20:37 |
| jogo | derekh: so I think we have/had that bug as a fingerprint already | 20:37 |
| *** dprince has quit IRC | 20:37 | |
| jogo | derekh: we have been using http://status.openstack.org/elastic-recheck/data/uncategorized.html for that in openstack | 20:37 |
| jogo | that is a list of unclassified gate failures. gate because we don't expect gate to fail ... versus check | 20:38 |
| Lotus907efi | hmmm, the django-openstack-auth python module seems to require a Django version newer than 1.4 and less than 1.6 but the run-mirror pulled in only 1.6.2 which did not satisfy the requirement | 20:38 |
| jogo | derekh: and its working pretty well for us | 20:39 |
| derekh | jogo: why isn't it reporting back to tripleo reviews? because it a defferent queue? | 20:39 |
| derekh | *different | 20:40 |
| jogo | derekh: not sure, right now its not reporting because ES is backed up | 20:40 |
| tchaypo | lifeless: if we fix the pip problem by having a local pypi mirror, doesn't that mean we'd have to tell anyone who wants to deploy with our tooling that setting up a 70Gb pypi mirror locally is a precondition for a working deploy? | 20:40 |
| jogo | way up | 20:40 |
| derekh | jogo: ok | 20:40 |
| jogo | queries/1272417.yaml | 20:41 |
| jogo | is the bug you just filed | 20:41 |
| derekh | 1292141.yaml | 20:42 |
| jogo | derekh: yours is a duplicate | 20:43 |
| *** jang1 has joined #tripleo | 20:43 | |
| jogo | is there a gate-tripleo queue? | 20:44 |
| jogo | or just check | 20:44 |
| derekh | jogo: ahh ok, just check | 20:45 |
| derekh | jogo: check-tripleo | 20:45 |
| derekh | lifeless: image built -rw-r--r-- 1 root root 619553792 Mar 19 20:38 /opt/stack/testenv-worker.qcow2 | 20:47 |
| greghaynes | tchaypo: pypi-mirror seems to work with only mirroring things we need | 20:48 |
| jogo | derekh: in that case the easiest way to figure out of things are down is graphite | 20:48 |
| greghaynes | tchaypo: mine is 193M | 20:48 |
| derekh | lifeless: I'm happy to test it later before we deploy system wide, but wont be able to for a few hours as I'll be dropping off soon | 20:48 |
| derekh | lifeless: or we could just deploy and keep our fingers crossed | 20:48 |
| jang1 | quite pleased with the latest pypi-mirror, by the way. Seems to iron out a lot of the bugs. | 20:48 |
| Lotus907efi | what was the env variable to set that would keep the cache? | 20:49 |
| greghaynes | USE_CACHE | 20:49 |
| bnemec | You have to pass -c to devtest.sh | 20:49 |
| greghaynes | its not 'keep the cache' | 20:49 |
| bnemec | It resets USE_CACHE to 0 | 20:49 |
| Lotus907efi | ok, so just set that to 1 and export before running devtest ? | 20:49 |
| Lotus907efi | oh | 20:49 |
| Lotus907efi | ok | 20:49 |
| greghaynes | just if it finds the image on disk it uses it rather than rebuilding | 20:49 |
| derekh | jogo: ahh is graphite running somewhere ? | 20:49 |
| * bnemec just made that mistake :-) | 20:49 | |
| tchaypo | greghaynes: but if I've understood correctly, lifeless is suggesting we need to have a full pypi mirror local to our test env to get reliable CI | 20:50 |
| jang1 | ^^ that really, really helps | 20:50 |
| jang1 | by "full" - pypi-mirror makes one. You don't need the 80G bandersnatch variant | 20:50 |
| jogo | derekh: yup http://graphite.openstack.org/ | 20:51 |
| greghaynes | tchaypo: Id be surprised if full really means 'all the packages' | 20:51 |
| jogo | and you will want something like stats.zuul.pipeline.check-tripleo.job.check-tripleo-undercloud-precise.FAILURE | 20:51 |
| greghaynes | because... you shouldnt need that | 20:51 |
| greghaynes | just all the packages we use | 20:51 |
| derekh | jogo: cool, thanks I'll give it a whirl and see what happens | 20:52 |
| tchaypo | The problems being talked about above where pip failing to get the index because it times out after 15 seconds. afaik pypi-mirror doesn't create an index. | 20:52 |
| jogo | you can do fancy graphite tricks and get failure percentege | 20:52 |
| jogo | and do something like this: jogo.github.io/gate' | 20:52 |
| jogo | jogo.github.io/gate | 20:52 |
| tchaypo | but maybe the solution is that pypi-mirror should locally cache an index and we pass that to pip? | 20:52 |
| Lotus907efi | my mirror here is only 330MB | 20:52 |
| *** rpodolyaka1 has joined #tripleo | 20:53 | |
| *** e0ne has quit IRC | 20:53 | |
| jang1 | have you got everything up and working now, Lotus907efi? | 20:53 |
| Lotus907efi | my mirror is being referenced | 20:53 |
| *** vkozhukalov_ has quit IRC | 20:54 | |
| *** e0ne has joined #tripleo | 20:54 | |
| Lotus907efi | once in a while it points up missing packages though | 20:54 |
| Lotus907efi | I just had an issue with a version problem on Django package | 20:54 |
| Lotus907efi | and had to go get an older version | 20:54 |
| Lotus907efi | to meet the requirements of a package that was referencing Django | 20:55 |
| jang1 | rather than suffer the death of a thousand cuts over the next n weeks, I encouraged everyone in the office to front-load the pestering for help over the last couple of days. | 20:56 |
| jang1 | my god, but we've seen some bizarre things. | 20:56 |
| jang1 | my favourite was the devtest_seed run that just stopped dead (bash hung) after build-seed-vm.sh returned. | 20:56 |
| jang1 | I have honestly no idea _why_ it did that, although we eventually caught an errant ssh process that seemed to trigger it. | 20:57 |
| *** rpodolyaka1 has quit IRC | 20:57 | |
| *** e0ne has quit IRC | 20:57 | |
| jang1 | also occasionally seeing tmpfses that aren't being cleaned up - looks like various system processes get their cwds into it. | 20:57 |
| SpamapS | jang1: there are some tmpfs leaks | 20:58 |
| SpamapS | today I feel like there is a brain leak.. can't seem to get my head focused | 20:58 |
| *** epim has quit IRC | 20:58 | |
| *** nati_ueno has joined #tripleo | 21:01 | |
| openstackgerrit | Chris Krelle proposed a change to openstack/tripleo-image-elements: Add Ironic config.json to seed-stack-config element https://review.openstack.org/81627 | 21:01 |
| tchaypo | jang1: was that your work on pull-tools that I patched over yesterday? | 21:01 |
| tchaypo | jang1: Does my patch still do what you intended? | 21:02 |
| *** epim has joined #tripleo | 21:03 | |
| jogo | derekh: http://jogo.github.io/gate/tripleo.html | 21:03 |
| lifeless | jang1: re errant ssh process | 21:03 |
| jogo | check data is very very dirty though | 21:03 |
| jogo | but that gives you an idea | 21:04 |
| lifeless | jang1: so ssh won't disconnect if it's fds are not closed | 21:04 |
| * tchaypo facepalms | 21:04 | |
| openstackgerrit | Chris Krelle proposed a change to openstack/tripleo-incubator: Add Ironic bits to _undercloud https://review.openstack.org/72969 | 21:04 |
| jang1 | that wasn't the problem here. literally, bash hung as boot-seed-vm exited and didnt' get as far as assigning to SEED_IP. | 21:04 |
| jang1 | I have honestly never seen anything like ti | 21:05 |
| tchaypo | Google calendar has a timezone labelled "(GMT +00:00) GMT (no daylight savings)" | 21:05 |
| lifeless | jang1: so you ssh to a host and run foo, foo gets stdout back to ssh, if foo blocks for some reason then the ssh process will hang | 21:05 |
| lifeless | jang1: ok | 21:05 |
| tchaypo | I think that possibly means "UTC", which is sometimes one hour different from GMT? | 21:05 |
| jang1 | GMT is UTC (only not sidereal, or what have you) | 21:06 |
| jang1 | british DST isn't called GMT | 21:07 |
| derekh | jogo: thanks, need to do some graphite research | 21:07 |
| NobodyCam | greghaynes: i'm waiting to see if the new https://review.openstack.org/#/c/72969 takes care of the key issue | 21:07 |
| derekh | tchaypo: I thought GMT and UTC were always within seconds of each other | 21:07 |
| jogo | you can see the query I used if you copy the image url | 21:08 |
| *** lazy_prince has joined #tripleo | 21:08 | |
| *** lazy_prince is now known as killer_prince | 21:08 | |
| jogo | and replace /render/ with /graphlot/ | 21:08 |
| jogo | or or just /plot/ | 21:09 |
| tchaypo | damn jang1 and derekh, you've spoiled my rant. | 21:09 |
| tchaypo | hrmph. | 21:09 |
| tchaypo | Fortunately this means I can move the target of my rant to the timezone labelled "(GMT +00:00) London" | 21:10 |
| tchaypo | which is self-contradictory, because London isn't always GMT +0 | 21:10 |
| tchaypo | I always feel those labels should either say "London" or "GMT+0", putting both in is confusing. | 21:11 |
| tchaypo | especially when you have places like eastern australia, where Google lists "GMT+10 Brisbane", "GMT+10 Melbourne/Sydney" and "GMT+10 Hobart" | 21:11 |
| *** martyntaylor has joined #tripleo | 21:12 | |
| *** blamar is now known as blamar-away | 21:12 | |
| * tchaypo drinks more coffee | 21:12 | |
| Lotus907efi | how long should a undercloud create process take? | 21:12 |
| Lotus907efi | I am at: Waiting for the undercloud stack to be ready | 21:13 |
| Lotus907efi | + wait_for 220 10 stack-ready undercloud | 21:13 |
| jang1 | the DIB part or the heat part? | 21:13 |
| tchaypo | Lotus907efi: I find that the check there isn't particularly good at catching failures. I usually jump onto the seed and keep an eye on "heat event-list undercloud" so I find out about failures early. | 21:13 |
| Lotus907efi | heat stack-list says the status is "CREATE_IN_PROCESS" | 21:13 |
| jang1 | it takes "a while" - you can log onto the seed and see where it's got to if you want. Or even log into the undercloud, once it's up, and watch the progress from within | 21:13 |
| jang1 | I think the undercloud standup (heat aprt) took about 300-350 seconds on my desktop today | 21:14 |
| *** hashar has quit IRC | 21:14 | |
| *** lblanchard has quit IRC | 21:14 | |
| Lotus907efi | hmmm | 21:15 |
| Lotus907efi | heat event-list undercloud is interesting | 21:15 |
| jang1 | what about nova list? is the undercloud actually active yet? | 21:15 |
| Lotus907efi | one small problem the person who first setup this system and environment put a passphrase on the ssh key | 21:16 |
| Lotus907efi | so once in a while it sits there waiting for the damn passphrase to be input | 21:16 |
| Lotus907efi | so I don't know if that is the problem now or not | 21:16 |
| Lotus907efi | undercloud is ACTIVE | 21:17 |
| Lotus907efi | (it was sitting there waiting for the passphrase) ARGH! | 21:17 |
| clarkb | replace the key? | 21:18 |
| Lotus907efi | yeah, but if I do that will I have to start from scratch again? | 21:20 |
| *** noslzzp has quit IRC | 21:21 | |
| openstackgerrit | Ben Nemec proposed a change to openstack/diskimage-builder: Set +x on executable files https://review.openstack.org/81633 | 21:22 |
| openstackgerrit | Ben Nemec proposed a change to openstack/diskimage-builder: Add dib-lint script https://review.openstack.org/81634 | 21:22 |
| openstackgerrit | Ben Nemec proposed a change to openstack/diskimage-builder: Alphabetize all element-deps https://review.openstack.org/81635 | 21:22 |
| openstackgerrit | Ben Nemec proposed a change to openstack/diskimage-builder: Enforce alphabetical ordering of element-deps https://review.openstack.org/81636 | 21:22 |
| openstackgerrit | Ben Nemec proposed a change to openstack/diskimage-builder: WIP: Make sure all scripts are set -e https://review.openstack.org/81637 | 21:22 |
| clarkb | I wouldn't expect so if you strip the passphrase off the key | 21:22 |
| clarkb | that way all of the public keys will still match | 21:22 |
| clarkb | but I am not familiar enough with devtest to know for sure | 21:22 |
| jang1 | it should certainly work, yeah | 21:22 |
| tchaypo | ssh-keygen -N will let you change the passphrase | 21:22 |
| tchaypo | so you shouldn't need to change anything since you'll still be using the same key | 21:23 |
| tchaypo | as clarkb already said | 21:23 |
| derekh | lifeless: new image in /opt/stack/testenv-worker.qcow2 I'll be back later to test it if (unless you deploy it first) | 21:23 |
| *** derekh has quit IRC | 21:24 | |
| bnemec | Are rechecks going to work right now or do we need to wait for the new testenv derekh was working on? | 21:31 |
| lifeless | bnemec: CI should be up right now | 21:36 |
| lifeless | I have a dr appt and will after that tear things down and redeploy the testenvs | 21:36 |
| bnemec | Cool. Didn't want to fire off rechecks if they were just going to fail again. | 21:36 |
| bnemec | Thanks | 21:36 |
| lifeless | bnemec: why did they fail ? | 21:36 |
| *** lifeless changes topic to "tripleo-cd running preserve-ephemeral WIP patches and https://review.openstack.org/#/c/62042/ | Using OpenStack to deploy OpenStack;meetings Tuesday 1900 UTC in #openstack-meeting-alt" | 21:37 | |
| bnemec | lifeless: Not sure, but something appears to have been hosed in the test environment: http://logs.openstack.org/66/74866/9/check-tripleo/check-tripleo-seed-precise/a89e2e5/console.html | 21:37 |
| lifeless | Lotus907efi: wow the ps -fH's really make that log hard to read | 21:38 |
| Lotus907efi | oh? | 21:38 |
| lifeless | Lotus907efi: yeah, its about 80% ps -fH output :( | 21:38 |
| Lotus907efi | ah | 21:38 |
| Lotus907efi | is that normal for that to be in there so much? | 21:38 |
| lifeless | we don't do that at all | 21:39 |
| lifeless | it must be a thing your colleague did | 21:39 |
| lifeless | "Michey ps -fH" | 21:39 |
| Lotus907efi | hmmm | 21:39 |
| Lotus907efi | weird | 21:39 |
| lifeless | ok so this: | 21:40 |
| lifeless | 10-os-collect-config 531.881 | 21:40 |
| lifeless | thats kindof long | 21:40 |
| lifeless | and | 21:40 |
| lifeless | 01-os-refresh-config 153.114 | 21:40 |
| lifeless | and wow | 21:41 |
| lifeless | 76-neutron 1201.215 | 21:41 |
| lifeless | so this looks like either IO performance on the local machine, or network performance, to me | 21:41 |
| lifeless | where is there machine, relative to yours | 21:41 |
| lifeless | to your mirror, I mean | 21:41 |
| lifeless | Lotus907efi: ^ | 21:42 |
| lifeless | Lotus907efi: I have to run to a dr appt, but I am speculating the answer is 'half way around the world' or something :) | 21:42 |
| Lotus907efi | it is in the same DC but in a different rack | 21:42 |
| openstackgerrit | Ben Nemec proposed a change to openstack/diskimage-builder: Alphabetize all element-deps https://review.openstack.org/81635 | 21:42 |
| Lotus907efi | both are in USeast | 21:42 |
| openstackgerrit | Ben Nemec proposed a change to openstack/diskimage-builder: WIP: Make sure all scripts are set -e https://review.openstack.org/81637 | 21:42 |
| openstackgerrit | Ben Nemec proposed a change to openstack/diskimage-builder: Enforce alphabetical ordering of element-deps https://review.openstack.org/81636 | 21:42 |
| lifeless | this is what I'd expect to see: | 21:42 |
| lifeless | 76-neutron 17.582 | 21:42 |
| Lotus907efi | hmmm | 21:42 |
| lifeless | gotta bounce, back in ~ 1 hr | 21:43 |
| * bnemec doesn't know his abc's | 21:43 | |
| Lotus907efi | where are you seeing those figures? | 21:43 |
| Lotus907efi | tht is something I can grep for in the logs? | 21:43 |
| jang1 | the profiling output, I'd expect. | 21:43 |
| Lotus907efi | hmmm | 21:43 |
| jang1 | dib dumps the time each script snippet took | 21:44 |
| Lotus907efi | I would like to compare those figures with the ones from my system that took a LOT less time | 21:44 |
| Lotus907efi | ok | 21:44 |
| Lotus907efi | hmm, my times seem a LOT less | 21:45 |
| jang1 | so there's either a network problem - trouble talking to a proxy, perhaps. | 21:45 |
| *** jtomasek has quit IRC | 21:45 | |
| jang1 | or, there's something else eating up all the disk bandwidth. | 21:45 |
| Lotus907efi | like 1/25 | 21:46 |
| Lotus907efi | hmmm | 21:46 |
| jang1 | I don't think pip behaves particularly well in the face of dropped connections (it's not marvellously robust in that respect) | 21:46 |
| Lotus907efi | for instance my time for 10-os-collect-config was on the order of 20.066 while her's was 531.881 | 21:46 |
| Lotus907efi | that is a huge diff | 21:46 |
| openstackgerrit | Chris Krelle proposed a change to openstack/tripleo-incubator: Add Ironic in to seed vm https://review.openstack.org/81643 | 21:46 |
| jang1 | so if the network's behaving oddly, it's not behaving oddly enough to cause pip to error out | 21:47 |
| jang1 | ooh ^^ that's an interesting change | 21:47 |
| *** ccrouch has quit IRC | 21:51 | |
| *** rpodolyaka1 has joined #tripleo | 21:53 | |
| Lotus907efi | yeah, is ironic far enough along to do that? | 21:54 |
| jang1 | it's further along than I'd had the impression it was prior to the mid-session hackathon. | 21:55 |
| jang1 | and, there's no better way to iron out the bugs than to use it in anger | 21:55 |
| *** rpodolyaka1 has quit IRC | 21:57 | |
| Lotus907efi | true | 21:59 |
| *** epim has quit IRC | 21:59 | |
| bnemec | Chris said he was able to successfully run devtest earlier today with Ironic. | 22:01 |
| *** epim has joined #tripleo | 22:02 | |
| *** e0ne has joined #tripleo | 22:04 | |
| *** e0ne has quit IRC | 22:08 | |
| *** martyntaylor has quit IRC | 22:08 | |
| *** epim has quit IRC | 22:09 | |
| openstackgerrit | Jenkins proposed a change to openstack/os-collect-config: Updated from global requirements https://review.openstack.org/81648 | 22:09 |
| openstackgerrit | Chris Krelle proposed a change to openstack/tripleo-incubator: Add Ironic in to seed vm https://review.openstack.org/81643 | 22:13 |
| *** spzala has joined #tripleo | 22:13 | |
| Lotus907efi | if I have a devtest run that completed on a system successfully so the seed, undercloud, and overcloud VMs are all up, if I want to do another run while preserving the cache (using -c option) will the earlier VMs be shutdown and cleaned up properly? | 22:15 |
| jang1 | yes | 22:15 |
| NobodyCam | bnemec: yep I was able to use Ironic to deploy undercloud from seed | 22:19 |
| *** bauzas has joined #tripleo | 22:19 | |
| tchaypo | Lotus907efi: If you're going to do it with devtest.sh make sure you pass --existing-environment | 22:23 |
| tchaypo | that will make it not skip _testenv.sh and just use the existing VMs | 22:23 |
| tchaypo | _testenv.sh performs poorly if there are already existing VMs - it doesn't create them, then it tramples on $TE_DATAFILE with its new information about not having any VMs | 22:24 |
| tchaypo | the alternative is to run cleanup-env, which will remove the VMs and make it safe to run _testenv.sh again | 22:25 |
| openstackgerrit | Chris Krelle proposed a change to openstack/tripleo-incubator: Add Ironic in to seed vm https://review.openstack.org/81643 | 22:25 |
| Lotus907efi | lifeless: did you ever get a reply from Rocky about the jq issue? | 22:27 |
| *** fandi has quit IRC | 22:28 | |
| *** jang1 has quit IRC | 22:30 | |
| *** howleyt has joined #tripleo | 22:33 | |
| Lotus907efi | hmmm | 22:35 |
| Lotus907efi | but doesn't cleanup-env also remove the cached images? | 22:35 |
| tchaypo | dunno *reads source* | 22:37 |
| tchaypo | doesn't look like it. | 22:37 |
| tchaypo | virsh destroy, virsh undefine, virsh vol-delete is all I'm seeing | 22:37 |
| *** epim has joined #tripleo | 22:43 | |
| *** CaptTofu has quit IRC | 22:49 | |
| *** CaptTofu has joined #tripleo | 22:50 | |
| *** yfaykya has joined #tripleo | 22:51 | |
| lifeless | Lotus907efi: no reply so far | 22:51 |
| Lotus907efi | hmmm | 22:52 |
| lifeless | Lotus907efi: yes. look for PROFILING | 22:52 |
| Lotus907efi | I can walk over to building 6 again and needle him | 22:52 |
| lifeless | jang: Lotus907efi: note that DIB tries to build in a tmpfs if possible to avoid all disk write IO issues (syncs, seeks etc) | 22:52 |
| Lotus907efi | ah, ok | 22:52 |
| Lotus907efi | lifeless: did you see my reply that this system and my pypi mirror system are in USeast but separate racks? | 22:53 |
| lifeless | Lotus907efi: I did | 22:54 |
| *** CaptTofu has quit IRC | 22:54 | |
| howleyt | Hi all | 22:54 |
| *** rpodolyaka1 has joined #tripleo | 22:54 | |
| lifeless | Lotus907efi: does it have a distro mirror as well to use ? | 22:55 |
| Lotus907efi | yes | 22:55 |
| lifeless | ok | 22:55 |
| yfaykya | howleyt: yo | 22:55 |
| lifeless | looking through the log for dib-run-parts Wed Mar 19 00:31:19 UTC 2014 Running /tmp/in_target.d/install.d/76-neutron | 22:55 |
| Lotus907efi | those accesses are pretty fast | 22:55 |
| lifeless | in more detail | 22:55 |
| Lotus907efi | I am watching the outout now | 22:55 |
| Lotus907efi | output | 22:55 |
| Lotus907efi | but the access to my pypi mirror seem slower | 22:55 |
| howleyt | I want to try bringing up a second notCompute node in my overcloud. What's the easiest way to do this? | 22:56 |
| lifeless | howleyt: you'll need to change the overcloud-source.yaml, I *think* greghaynes might have a draft of such changes around | 22:57 |
| howleyt | Is it a case of modifying overcloud-source.yaml to duplicate... | 22:57 |
| lifeless | howleyt: no, you'll want a SCALE variable approach like we do for compute | 22:57 |
| lifeless | howleyt: see Makefile | 22:57 |
| greghaynes | hrm, I just copied the notcompute0 and s/notcompute0/notcompute1 in the destination block essentially | 22:58 |
| howleyt | lifeless: that sounds better | 22:58 |
| lifeless | howleyt: we've done some ground work for scaling notCompute but it was/is blocked on having HA ready plumbing | 22:58 |
| lifeless | greghaynes: oh you bad boy :) | 22:58 |
| greghaynes | haha | 22:58 |
| howleyt | Ya, I started doing that and it felt bad.. | 22:58 |
| greghaynes | Yes, not sure its something I should really share | 22:58 |
| tchaypo | Please share | 22:59 |
| *** rpodolyaka1 has quit IRC | 22:59 | |
| tchaypo | It'll make the rest of us feel better about ourselves : | 22:59 |
| tchaypo | :) | 22:59 |
| greghaynes | I should probably mess with merge.py'ifying it now that I kind of understand whats going on.... | 23:00 |
| tchaypo | i hate the "notCompute" name. | 23:02 |
| howleyt | So the merge script wouldn't necessarily handle a --scale NotCompute as it stands? | 23:02 |
| greghaynes | tchaypo: So say we all | 23:02 |
| greghaynes | howleyt: it certainly doesnt | 23:02 |
| greghaynes | howleyt: pretty easy to test that :) | 23:02 |
| howleyt | indeed (on the naming) | 23:03 |
| *** yamahata has quit IRC | 23:04 | |
| greghaynes | I wonder why we dont call it control... seems like we use that name elsewhere | 23:05 |
| greghaynes | or maybe thats exactly why we dont call it control | 23:05 |
| howleyt | greghaynes: I guess working on a copy as you did could work, for starters. Just wanted to check in case I was missing something. | 23:05 |
| SpamapS | graaahhh.. why does vim insist on over-indenting yaml whenever I type something: ? | 23:06 |
| greghaynes | howleyt: yes, the issue is that running multiple nodes doesnt work as is not because of the template but because of rabbit and mysql not being able to handle it | 23:06 |
| greghaynes | as well as some state needing to be moved into the db AIUI | 23:07 |
| tchaypo | greghaynes: or rather, because we're not supplying configs to rabbit/mysql to tell them about the fact that they live in a cluster | 23:07 |
| tchaypo | right? | 23:07 |
| *** nati_ueno has quit IRC | 23:07 | |
| howleyt | So I'm interested in trying out the rabbit cluster element changes that have landed. | 23:07 |
| greghaynes | that might be true for rabbit after yesterdays patches, not true for mysql yet | 23:07 |
| lifeless | Lotus907efi: so we need more timestamps from e.g. pip. but yes, we have ok mirror perf - Fetched 339 kB in 0s (1681 kB/s) | 23:08 |
| lifeless | for linux packages | 23:08 |
| tchaypo | SpamapS: because you don't have filetype-specific ts/sw settings? | 23:08 |
| howleyt | greghaynes: understood | 23:08 |
| Lotus907efi | hmmm, ok | 23:08 |
| *** bauzas has quit IRC | 23:08 | |
| greghaynes | howleyt: jprovazn is the person to chat with on that. You might be able to bake up a template that only clusters rabbit on the !0 notcompute nodes... | 23:08 |
| SpamapS | tchaypo: it does it _while I type_ | 23:08 |
| Lotus907efi | how to get more time stamps from pip? | 23:09 |
| SpamapS | tchaypo: I presume this is 'autoindent' .. but I set noautoindent.. so... graahh | 23:09 |
| howleyt | greghaynes: Will do. So I'd have to make sure that mysql is only deployed on notcompute0 so that everything else works happily? | 23:10 |
| greghaynes | My inexperienced mind thinks that might kind of work... I am still figuring this out as well though | 23:11 |
| lifeless | Lotus907efi: not sure :) | 23:11 |
| NobodyCam | lifeless: just checking befor I start this is the in progress work I need to rebase off of: https://review.openstack.org/#/c/78467 | 23:11 |
| * greghaynes looks around for others to confirm/deny that idea | 23:11 | |
| *** newell has quit IRC | 23:11 | |
| lifeless | NobodyCam: I think so yes. StevenK - I'm suggesting NobodyCam work with your nodes refactoring stuff | 23:11 |
| tchaypo | howleyt: that sounds feasbile.. but if mysql is only running on one node, why are you running up multiple nodes? | 23:12 |
| greghaynes | tchaypo: just to test the rabbit parts I think | 23:12 |
| howleyt | tchaypo: I want to try out rabbitmq cluster | 23:12 |
| tchaypo | ah, very good. | 23:12 |
| SpamapS | aha | 23:13 |
| greghaynes | ok, enough internal videos for the week... time to code | 23:13 |
| SpamapS | http://vim.wikia.com/wiki/How_to_stop_auto_indenting | 23:13 |
| SpamapS | fixed it | 23:13 |
| *** cwolferh_ has quit IRC | 23:14 | |
| *** derekh has joined #tripleo | 23:15 | |
| *** sseago has joined #tripleo | 23:17 | |
| lifeless | howleyt: so, what you could do is make a template that just configures rabbit on two nodes and build an image with just rabbit on it | 23:18 |
| openstackgerrit | Clint "SpamapS" Byrum proposed a change to openstack/tripleo-heat-templates: WIP: Really ugly first switch to software-config https://review.openstack.org/81666 | 23:18 |
| *** weshay has quit IRC | 23:19 | |
| SpamapS | stevebaker: ^^ there are a million things wrong.. but I have made "progress" | 23:19 |
| stevebaker | SpamapS: does anything actually work? | 23:21 |
| Lotus907efi | hmmm, overcloud CREATE_FAILED | 23:22 |
| Lotus907efi | overcloud-notCompute0-mdwo3l7idqo7 | ERROR | 23:22 |
| Lotus907efi | but overcloud-NovaCompute0-k55pidjqgfn5 | ACTIVE | RUNNING | 23:22 |
| Lotus907efi | and overcloud-NovaCompute1-nijjq5jmxx3o | ACTIVE | Running | 23:22 |
| SpamapS | stevebaker: no | 23:23 |
| SpamapS | stevebaker: need to fix merge.py a little more, or the templates.. not sure | 23:23 |
| SpamapS | oops, added a .tmp file | 23:24 |
| SpamapS | anyway.. | 23:25 |
| * SpamapS -> kids | 23:25 | |
| stevebaker | SpamapS: looking now | 23:25 |
| *** cwolferh_ has joined #tripleo | 23:29 | |
| * greghaynes tries to learn what software-config is about | 23:29 | |
| *** epim has quit IRC | 23:30 | |
| stevebaker | greghaynes: any questions let me know | 23:30 |
| greghaynes | Any docs you know of? At the moment im just reading tests | 23:30 |
| stevebaker | greghaynes: I'm writing them now. There is this https://review.openstack.org/#/c/81407/ https://review.openstack.org/#/c/81629/ https://review.openstack.org/#/q/project:openstack/heat-templates+topic:software_config,n,z | 23:31 |
| greghaynes | aha! | 23:32 |
| *** epim has joined #tripleo | 23:37 | |
| *** spzala has quit IRC | 23:38 | |
| *** spzala has joined #tripleo | 23:38 | |
| *** spzala has quit IRC | 23:39 | |
| tchaypo | I wonder if HP has a corp account with Safari | 23:39 |
| *** spzala has joined #tripleo | 23:40 | |
| *** epim has quit IRC | 23:42 | |
| howleyt | lifeless: sounds like a good plan also, might force me learn what is actually happening with heat and the yaml configs. | 23:43 |
| howleyt | I'm going to sign off, folks, bedtime for me. Thanks for the help. | 23:45 |
| lifeless | night | 23:46 |
| *** howleyt has quit IRC | 23:47 | |
| *** epim has joined #tripleo | 23:51 | |
| *** rpodolyaka1 has joined #tripleo | 23:55 | |
| *** rpodolyaka1 has quit IRC | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!