| tchaypo | ssh works over the pacific. vnc requires more bandwidth. | 00:11 |
|---|---|---|
| tchaypo | but I'm increasingly of the opinion that the only real benefit here is that I've historically been used to virsh console | 00:12 |
| tchaypo | I actually used vnc just fine from the HP cafeteria. The amount of mental effort I'm throwing into this when there's already an 85% solution... | 00:12 |
| lifeless | tchaypo: so in that review, I wasn't objecting to the feature, just trying to shape it to be a bit cleaner. That said, yeah. vnc :)_ | 00:13 |
| *** ramishra has quit IRC | 00:15 | |
| *** sseago has quit IRC | 00:16 | |
| *** ccrouch has joined #tripleo | 00:17 | |
| lifeless | tchaypo: you do know that vnc was *designed* for low bw situations? :> | 00:18 |
| lifeless | it was competing with X... | 00:18 |
| lifeless | and ICA/RDP | 00:18 |
| Ng | vnc was designed for local networks! | 00:18 |
| tchaypo | yeah, understood. I'm not giving up because it's too hard, I'm deciding it's not a good investment of time any more | 00:18 |
| *** sseago_ has joined #tripleo | 00:18 | |
| lifeless | tchaypo: ack | 00:18 |
| lifeless | Ng: 1Mbps is sufficient though | 00:18 |
| *** ccrouch has quit IRC | 00:18 | |
| *** ccrouch has joined #tripleo | 00:18 | |
| *** ramishra has joined #tripleo | 00:19 | |
| *** ramishra has quit IRC | 00:19 | |
| *** ramishra has joined #tripleo | 00:19 | |
| openstackgerrit | Dan Prince proposed a change to openstack/tripleo-image-elements: TEST ONLY: make nova depend on common-venv https://review.openstack.org/79989 | 00:19 |
| openstackgerrit | Dan Prince proposed a change to openstack/tripleo-image-elements: Openstack-clients: don't hard code venv https://review.openstack.org/79988 | 00:19 |
| openstackgerrit | Dan Prince proposed a change to openstack/tripleo-image-elements: Update elements to use os-svc-daemon -i. https://review.openstack.org/76963 | 00:19 |
| openstackgerrit | Dan Prince proposed a change to openstack/tripleo-image-elements: Update elements to use os-svc-install -i. https://review.openstack.org/76964 | 00:19 |
| openstackgerrit | Dan Prince proposed a change to openstack/tripleo-image-elements: Wire in OPENSTACK_VENV_DIR default if set https://review.openstack.org/76965 | 00:19 |
| openstackgerrit | Dan Prince proposed a change to openstack/tripleo-image-elements: Wire in _EXTRA_INSTALL_OPTS... https://review.openstack.org/76966 | 00:19 |
| openstackgerrit | Dan Prince proposed a change to openstack/tripleo-image-elements: Heat/Horizon/Tempest: don't hard code venv dir https://review.openstack.org/76888 | 00:19 |
| openstackgerrit | Dan Prince proposed a change to openstack/tripleo-image-elements: Add a new common-venv element https://review.openstack.org/76967 | 00:19 |
| openstackgerrit | Dan Prince proposed a change to openstack/tripleo-image-elements: Glance/Cinder/Keystone/Swift: don't hard code venv https://review.openstack.org/76666 | 00:19 |
| openstackgerrit | Dan Prince proposed a change to openstack/tripleo-image-elements: Ceilo/Ironic/Tuskar: don't hard code venv dir https://review.openstack.org/76887 | 00:19 |
| openstackgerrit | Dan Prince proposed a change to openstack/tripleo-image-elements: Add -i options to os-svc-install/os-svc-daemon... https://review.openstack.org/76952 | 00:19 |
| lifeless | ruh roh | 00:20 |
| Ng | gah, I was half way up that stack | 00:20 |
| * Ng abandons ship for the calm seas of sleep | 00:22 | |
| *** d0ugal has quit IRC | 00:26 | |
| *** d0ugal has joined #tripleo | 00:27 | |
| *** d0ugal has quit IRC | 00:27 | |
| *** d0ugal has joined #tripleo | 00:27 | |
| *** matsuhashi has joined #tripleo | 00:31 | |
| *** ramishra has quit IRC | 00:31 | |
| *** Lotus907efi is now known as Lotus907efi- | 00:36 | |
| *** ccrouch1 has joined #tripleo | 00:48 | |
| *** openstack has joined #tripleo | 00:53 | |
| *** bogdando has joined #tripleo | 00:54 | |
| *** bcrochet_ has joined #tripleo | 00:55 | |
| *** bcrochet_ has quit IRC | 00:55 | |
| *** bcrochet_ has joined #tripleo | 00:55 | |
| *** bcrochet_ is now known as bcrochet | 00:55 | |
| *** openstack has joined #tripleo | 00:57 | |
| *** openstack has quit IRC | 00:57 | |
| *** openstack has joined #tripleo | 01:00 | |
| *** blamar_ has joined #tripleo | 01:02 | |
| *** rbrady-afk has quit IRC | 01:02 | |
| *** SpamapS has quit IRC | 01:04 | |
| *** blamar has quit IRC | 01:06 | |
| *** rpodolyaka1 has joined #tripleo | 01:07 | |
| *** matsuhashi has quit IRC | 01:08 | |
| *** SpamapS_ has joined #tripleo | 01:08 | |
| *** rpodolyaka has quit IRC | 01:08 | |
| *** blamar_ is now known as blamar | 01:08 | |
| *** julim has quit IRC | 01:12 | |
| *** matsuhashi has joined #tripleo | 01:12 | |
| *** rpodolyaka1 has quit IRC | 01:14 | |
| *** openstack has joined #tripleo | 01:15 | |
| *** openstack has joined #tripleo | 01:19 | |
| *** nosnos has joined #tripleo | 01:19 | |
| *** openstack has quit IRC | 01:24 | |
| *** openstack has joined #tripleo | 01:25 | |
| *** openstack has joined #tripleo | 15:09 | |
| openstackgerrit | Ryan Brady proposed a change to openstack/tripleo-heat-templates: Remove unnecessary cinder metadata https://review.openstack.org/80600 | 15:09 |
| openstackgerrit | Radomir Dopieralski proposed a change to openstack/tuskar-ui: Node profile suggestions https://review.openstack.org/78563 | 15:12 |
| openstackgerrit | A change was merged to openstack/tuskar-ui: Fix error 500 on deployment design form validation https://review.openstack.org/80542 | 15:15 |
| openstackgerrit | Chris Krelle proposed a change to openstack/tripleo-heat-templates: Change nova compute driver location https://review.openstack.org/80601 | 15:15 |
| *** rpodolyaka has joined #tripleo | 15:15 | |
| openstackgerrit | Derek Higgins proposed a change to openstack-infra/tripleo-ci: Add hostname of Test env to json https://review.openstack.org/80603 | 15:18 |
| *** dprince has joined #tripleo | 15:18 | |
| *** rdopieralski has quit IRC | 15:21 | |
| *** sdake has joined #tripleo | 15:24 | |
| openstackgerrit | Jon-Paul Sullivan proposed a change to openstack/tripleo-incubator: Ensure a default value for DEVTEST_PERF_COMMENT https://review.openstack.org/80584 | 15:28 |
| *** andrearo1 is now known as andrearosa | 15:30 | |
| *** marun has quit IRC | 15:39 | |
| openstackgerrit | Chris Krelle proposed a change to openstack/tripleo-image-elements: Correct permissions on /tftpdboot/pxelinux.cfg directory https://review.openstack.org/80595 | 15:41 |
| *** rpodolyaka has quit IRC | 15:42 | |
| *** CaptTofu has quit IRC | 15:44 | |
| *** CaptTofu has joined #tripleo | 15:44 | |
| *** CaptTofu has quit IRC | 15:48 | |
| *** NobodyCam is now known as BlartVersenwald | 15:49 | |
| *** pblaho has quit IRC | 15:52 | |
| *** rbrady is now known as rbrady-afk | 16:01 | |
| *** d0ugal has quit IRC | 16:01 | |
| openstackgerrit | jan grant proposed a change to openstack/tripleo-incubator: Use the DIB_REPOLOCATION / DIB_REPOREF for the bootstrap. https://review.openstack.org/78371 | 16:01 |
| *** d0ugal has joined #tripleo | 16:04 | |
| *** d0ugal has joined #tripleo | 16:04 | |
| *** matty_dubs is now known as matty_dubs|lunch | 16:09 | |
| *** Kim-Chi-San has quit IRC | 16:12 | |
| *** Kim-Chi-San has joined #tripleo | 16:14 | |
| *** akrivoka has quit IRC | 16:15 | |
| *** Kim-Chi-San has quit IRC | 16:15 | |
| *** tburnes has joined #tripleo | 16:15 | |
| *** tburnes has joined #tripleo | 16:16 | |
| openstackgerrit | James Slagle proposed a change to openstack/diskimage-builder: Move install type enablement into base element https://review.openstack.org/78794 | 16:16 |
| openstackgerrit | Matthew Gilliard proposed a change to openstack/tripleo-incubator: Allow set-source-vars to consume a git manifest https://review.openstack.org/80617 | 16:22 |
| *** jang has joined #tripleo | 16:23 | |
| *** hewbrocca has quit IRC | 16:23 | |
| openstackgerrit | James Slagle proposed a change to openstack/diskimage-builder: Move install type enablement into base element https://review.openstack.org/78794 | 16:25 |
| *** rpodolyaka has joined #tripleo | 16:25 | |
| *** vkozhukalov_ has joined #tripleo | 16:26 | |
| rpodolyaka | morning all | 16:28 |
| *** bauzas has quit IRC | 16:30 | |
| openstackgerrit | Matthew Gilliard proposed a change to openstack/tripleo-incubator: Allow set-source-vars to consume a git manifest https://review.openstack.org/80617 | 16:30 |
| *** jistr has quit IRC | 16:31 | |
| openstackgerrit | tom-howley proposed a change to openstack/tripleo-incubator: WIP: Add standalone script to build all images. https://review.openstack.org/77462 | 16:33 |
| *** ccrouch has quit IRC | 16:35 | |
| *** martyntaylor has left #tripleo | 16:38 | |
| *** openstackstatus has joined #tripleo | 16:39 | |
| *** blamar has quit IRC | 16:41 | |
| *** blamar has joined #tripleo | 16:42 | |
| openstackgerrit | therese-mchale proposed a change to openstack/tripleo-image-elements: Add a softlink for /var/log/mysql https://review.openstack.org/80567 | 16:45 |
| *** ifarkas has quit IRC | 16:47 | |
| *** ifarkas has joined #tripleo | 16:47 | |
| *** giulivo has quit IRC | 16:50 | |
| *** rpodolyaka has quit IRC | 16:51 | |
| *** ccrouch has joined #tripleo | 16:53 | |
| *** akrivoka has joined #tripleo | 16:53 | |
| *** petertoft has quit IRC | 16:57 | |
| *** rpodolyaka has joined #tripleo | 16:58 | |
| openstackgerrit | A change was merged to openstack/tripleo-image-elements: Heat/Horizon/Tempest: don't hard code venv dir https://review.openstack.org/76888 | 17:03 |
| openstackgerrit | A change was merged to openstack/diskimage-builder: RHEL Package maps for build-essential, python-dev and libz-dev. https://review.openstack.org/79286 | 17:03 |
| openstackgerrit | tom-howley proposed a change to openstack/tripleo-incubator: WIP: Add standalone script to build all images. https://review.openstack.org/77462 | 17:04 |
| openstackgerrit | tom-howley proposed a change to openstack/tripleo-incubator: Add standalone script to build all images. https://review.openstack.org/77462 | 17:05 |
| *** matty_dubs|lunch is now known as matty_dubs | 17:06 | |
| *** funzo_ is now known as funzo | 17:06 | |
| openstackgerrit | Derek Higgins proposed a change to openstack/diskimage-builder: Create raw disk image outside of tmpfs https://review.openstack.org/79362 | 17:10 |
| *** pbelanyi_ has quit IRC | 17:12 | |
| SpamapS | o/ | 17:16 |
| *** rbrady-afk is now known as rbrady | 17:17 | |
| *** fandi has joined #tripleo | 17:17 | |
| *** hashar has quit IRC | 17:20 | |
| greghaynes | morning | 17:20 |
| *** morganfainberg_Z is now known as morganfainberg | 17:21 | |
| openstackgerrit | James Slagle proposed a change to openstack/tripleo-image-elements: os-*-config package install support https://review.openstack.org/78816 | 17:21 |
| *** e0ne has quit IRC | 17:22 | |
| *** spzala_ has joined #tripleo | 17:26 | |
| *** spzala_ has quit IRC | 17:26 | |
| *** spzala has quit IRC | 17:27 | |
| *** jp_at_hp1 has joined #tripleo | 17:28 | |
| beekneemech | Is there any chance I could get another core to look at https://review.openstack.org/#/c/74866/ and https://review.openstack.org/#/c/74867/ before another change merges to the fedora element and forces a rebase again? | 17:28 |
| *** jp_at_hp has quit IRC | 17:28 | |
| beekneemech | I think everyone was okay with the changes, but they've been stuck on rebases and ci issues. | 17:28 |
| openstackgerrit | Dan Prince proposed a change to openstack/diskimage-builder: Support adding DHCP interfaces one at a time. https://review.openstack.org/78711 | 17:30 |
| *** dprince has quit IRC | 17:31 | |
| *** noslzzp has quit IRC | 17:34 | |
| derekh | beekneemech: running it now, will check back later to see how it goes (don't trust ci as its fedora ...), unless of course somebody else merges it in the meantime | 17:37 |
| beekneemech | derekh: Sure, thanks. | 17:37 |
| derekh | anybody interested in keeping an eye on CI jobs and how they are doing overall http://goodsquishy.com/downloads/tripleo-jobs.html | 17:38 |
| *** athomas has quit IRC | 17:38 | |
| slagle | there's got to be a story behind that domain name | 17:42 |
| derekh | slagle: ya, I think at the time derekhiggins.com was gone :-( and I was watching the simpsons http://www.youtube.com/watch?v=yKRWV0yjqJ0 | 17:43 |
| *** asparks has quit IRC | 17:44 | |
| Lotus907efi | I finally got my pypi run-mirror to be a little more successful here on a machine in our data center | 17:44 |
| *** rpodolyaka has quit IRC | 17:45 | |
| Lotus907efi | but whenever I do the mirror it fails on building one module, httpretty | 17:45 |
| Lotus907efi | is that expected or particularly bad | 17:45 |
| Lotus907efi | or can I ignore that? | 17:45 |
| *** CaptTofu has joined #tripleo | 17:46 | |
| slagle | Lotus907efi: that's actually a known issue with the openstack pypi mirror right now | 17:47 |
| Lotus907efi | hmm, would it be ok to ignore it for right now or is there a way to fix it? | 17:48 |
| Lotus907efi | we need the mirror here so other people here can use it to do trippleo testing | 17:48 |
| Lotus907efi | in a DC without direct access to the internet | 17:48 |
| clarkb | slagle: what is the issue? | 17:48 |
| *** rpodolyaka has joined #tripleo | 17:48 | |
| clarkb | just httpretty? | 17:49 |
| Lotus907efi | yes | 17:49 |
| slagle | hold on, i'm trying to search for the bug | 17:49 |
| clarkb | last time I looked at our logs they were clean | 17:49 |
| Lotus907efi | that is the only one failing for me | 17:49 |
| slagle | i know one got filed | 17:49 |
| slagle | ah, my search was failing b/c i was thinking of PrettyTable | 17:50 |
| slagle | https://bugs.launchpad.net/tripleo/+bug/1292141 | 17:50 |
| slagle | clarkb: ^^ i don't honestly know if it's a problem with the mirror or not | 17:51 |
| clarkb | slagle: its not, iirc its a problem with your pip | 17:51 |
| *** jdob has quit IRC | 17:51 | |
| clarkb | though we could probably make the apache serve case insensitive | 17:52 |
| clarkb | pip will mostly be case insensitive but in some cases we haev seen it get picky | 17:52 |
| clarkb | iirc when wheels are enalbed on the 1.5 release | 17:52 |
| *** jdob has joined #tripleo | 17:52 | |
| slagle | something was odd, b/c we saw this exact same error a few times, but wasn't 100% reproduceable | 17:53 |
| clarkb | oh but latest comment says no amount of case insensitivity will help | 17:53 |
| * clarkb reads more | 17:53 | |
| slagle | so i don't think it was just a "network blip" | 17:53 |
| Lotus907efi | hmm, so is there a bug with httpretty as well or have I run into something unusual? | 17:53 |
| slagle | Lotus907efi: there isn't one that i know of | 17:54 |
| Lotus907efi | I can capture the output from the run-mirror execution here is that would be useful or illustrative to anyone | 17:54 |
| slagle | Lotus907efi: if it's reproduceable, it's certainly worth filing | 17:55 |
| clarkb | ah yes its complaining about being unable to get the index | 17:55 |
| clarkb | so insensitivity not to blame | 17:55 |
| Lotus907efi | also the run-mirror process hangs for a very long time during one module build part of the process | 17:55 |
| openstackgerrit | Jon-Paul Sullivan proposed a change to openstack/tripleo-image-elements: Create pip manifest files of installed packages https://review.openstack.org/80644 | 17:56 |
| Lotus907efi | like for 10 - 15 minutes | 17:56 |
| clarkb | it does quite a bit of downloading and installing | 17:56 |
| clarkb | to ensure that all packages are cached | 17:56 |
| *** derekh has quit IRC | 17:58 | |
| *** nati_ueno has joined #tripleo | 17:59 | |
| nati_ueno | Hi! | 17:59 |
| nati_ueno | I'm trying to use triple-o. I have two questions. | 17:59 |
| nati_ueno | Is there any debug hacks if the overcloud machines booted-up and they won't be able to ping ? | 18:00 |
| nati_ueno | also, where can I get latest docs? | 18:01 |
| Lotus907efi | docs are online in openstack wiki | 18:04 |
| Lotus907efi | are you able to get to the consoles of your overcloud systems with virt-manager? | 18:05 |
| *** marun has joined #tripleo | 18:05 | |
| *** e0ne has joined #tripleo | 18:11 | |
| *** jp_at_hp1 has quit IRC | 18:14 | |
| *** cwolferh_ has joined #tripleo | 18:16 | |
| *** cwolferh has quit IRC | 18:17 | |
| *** akrivoka has quit IRC | 18:18 | |
| nati_ueno | Lotus907efi: i'm using baremetal for controllers, so I can't use virt-manager | 18:20 |
| nati_ueno | Lotus907efi: so docs in the wiki is up-to-dated? | 18:20 |
| *** jdob has quit IRC | 18:25 | |
| *** rwsu has quit IRC | 18:30 | |
| Lotus907efi | nati_ueno: can you get to the consoles of the systems any other way? | 18:36 |
| *** rpodolyaka has quit IRC | 18:39 | |
| openstackgerrit | James Slagle proposed a change to openstack/tripleo-image-elements: Add fedora-rdo-icehouse element https://review.openstack.org/80658 | 18:40 |
| openstackgerrit | James Slagle proposed a change to openstack/tripleo-image-elements: Add fedora-rdo-icehouse element https://review.openstack.org/80658 | 18:42 |
| *** dprince has joined #tripleo | 18:43 | |
| *** ccrouch has quit IRC | 18:43 | |
| *** morazi has quit IRC | 18:43 | |
| *** morazi has joined #tripleo | 18:46 | |
| *** rpodolyaka has joined #tripleo | 18:47 | |
| Lotus907efi | nati_ueno: I am suggesting you somehow check the consoles of your systems to see if they actually booted a valid OS image or if they are somehow stuck somehwere int he boot process | 18:49 |
| Lotus907efi | stuck somewhere in the boot process | 18:49 |
| lifeless | slagle: why do you think it is not a network blip ? | 18:52 |
| slagle | lifeless: saw the same failure on 2 different ci jobs at different times | 18:54 |
| dprince | lifeless/SpamapS: have you guys seen issues with dhclient sending a DHCPDECLINE before? | 18:54 |
| dprince | lifeless/SpamapS: we are hitting this issue in the RH rack. | 18:54 |
| slagle | lifeless: could certainly be a coincidence i suppose. that just seemed unlikely | 18:54 |
| dprince | lifeless/SpamapS: essentially things come up perfectly, but after 24 hours the DHCP leases expire and dhclient sends out a DHCPDECLINE response which causes a loss of connectivity. | 18:55 |
| *** gcha has quit IRC | 18:57 | |
| *** morazi has quit IRC | 18:59 | |
| *** d0ugal has quit IRC | 19:00 | |
| *** jogo is now known as flashgordon | 19:03 | |
| *** jistr has joined #tripleo | 19:04 | |
| *** ccrouch has joined #tripleo | 19:04 | |
| *** jprovazn is now known as jprovazn_afk | 19:09 | |
| *** marun has quit IRC | 19:11 | |
| *** morazi has joined #tripleo | 19:11 | |
| *** rpodolyaka has quit IRC | 19:14 | |
| lifeless | dprince: on testenv of ci-overcloud machines? | 19:16 |
| *** rpodolyaka has joined #tripleo | 19:17 | |
| lifeless | testenv or ci-overcloud machines | 19:17 |
| dprince | lifeless: pretty much everywhere. I saw it on an undercloud machine (DHCP between the seed and undercloud). And we see the same thing on the overcloud machines. | 19:17 |
| dprince | lifeless: it happens to testenv's too | 19:17 |
| SpamapS | shouldn't dnclient be renewing well before lease expiration? | 19:17 |
| dprince | lifeless: it appears to be a DHCP issue where dhclient arps and thinks the address it already has is in use elsewhere. | 19:18 |
| dprince | lifeless: and then it sends back the DHCPDECLINE, which causes Neutron's dnsmasq server to decline further renew requests for like 10 minutes. | 19:18 |
| dprince | lifeless: total unhappyness, but possibely related to our network setup. I just wanted to ask to see if you guys ever hit anything similar | 19:19 |
| SpamapS | is this because we're dhcp'ing on the bridge? | 19:19 |
| SpamapS | like, perhaps we need to tell dhclient what its' real hwaddr is? | 19:21 |
| *** CaptTofu has quit IRC | 19:22 | |
| *** CaptTofu has joined #tripleo | 19:22 | |
| *** rpodolyaka has quit IRC | 19:25 | |
| *** rpodolyaka has joined #tripleo | 19:26 | |
| lifeless | dprince: never seend DHCPDECLINE | 19:26 |
| *** CaptTofu has quit IRC | 19:27 | |
| lifeless | dprince: thought it might be the dhcp reuse issue in neutron (sounds like not) or perhaps the dhcping with two addresses known issue for the testenvs | 19:27 |
| dprince | lifeless: cool. Thanks man. Just asking. Aside from this issue Derek and I have high confidence we are ready for the onslaught | 19:28 |
| lifeless | dprince: https://bugs.launchpad.net/tripleo/+bug/1272803 | 19:28 |
| lifeless | dprince: sweet | 19:28 |
| *** hashar has joined #tripleo | 19:28 | |
| lifeless | dprince: so if it arps and finds the same address in use elsewhere we should be able to capture that by tcpdumping traffic for that address - arp request and responses, on one of the boxes | 19:29 |
| dprince | lifeless: 1272803 isn't our issue here. | 19:29 |
| lifeless | dprince: yeah didn't think so from the symptoms | 19:33 |
| *** nijaba has quit IRC | 19:33 | |
| lifeless | dprince: so yeah, I'd put a background tcpdump running for arp | 19:37 |
| lifeless | dprince: and when the problem occurs, we can look at that | 19:38 |
| dprince | lifeless: so we think setting arp_ignore=1 may fix it | 19:38 |
| *** nijaba has joined #tripleo | 19:38 | |
| *** nijaba has quit IRC | 19:38 | |
| *** nijaba has joined #tripleo | 19:38 | |
| *** e0ne has quit IRC | 19:38 | |
| dprince | lifeless: i.e. reply only if the target IP address is local address configured on the incomming interface | 19:39 |
| *** e0ne has joined #tripleo | 19:39 | |
| lifeless | dprince: if we need that we have a broken host somewhere | 19:39 |
| lifeless | dprince: our stock topology should not result in false arps | 19:40 |
| dprince | lifeless: So in our rack things were initially configured for em1/em2 to be fully bonded. | 19:40 |
| *** nijaba has quit IRC | 19:40 | |
| dprince | lifeless: em1 goes to a switch, and em2 goes to different switch. But the two switch are connected | 19:41 |
| lifeless | they report the same switch id ? | 19:42 |
| lifeless | or different? | 19:42 |
| dprince | lifeless: Could be that, or something on the bastion | 19:42 |
| dprince | lifeless: not sure about the switch ID | 19:42 |
| SpamapS | lifeless: wouldn't dhclient think that its' MAC is the mac of the bridge, rather than the real interface? | 19:42 |
| lifeless | dprince: if we tcpdump we'll see the mac of the culprit thats arping back | 19:42 |
| lifeless | SpamapS: 6: br-ctlplane: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT | 19:43 |
| lifeless | link/ether 78:e7:d1:23:9d:7d brd ff:ff:ff:ff:ff:ff | 19:43 |
| lifeless | 50: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000 | 19:43 |
| lifeless | link/ether 78:e7:d1:23:9d:7d brd ff:ff:ff:ff:ff:ff | 19:43 |
| dprince | SpamapS: No, I'm using this https://review.openstack.org/#/c/69918/. Plus I've reproduce this without using an OVS bridge at all. Just plain old em1 dhcp. | 19:43 |
| SpamapS | right | 19:44 |
| *** e0ne_ has joined #tripleo | 19:44 | |
| lifeless | dprince: so it may indeed be that we need bonding configured to work properly in this setup. All i'm proposing is that we properly identify the cause before working around | 19:44 |
| SpamapS | what about the weak arp model of linux? Is that still an issue? (as in, arp packets that arrive on em1 for an ip on em2 are ack'd on em1) | 19:45 |
| dprince | SpamapS: that is exactly what I was thinking here | 19:45 |
| *** e0ne has quit IRC | 19:45 | |
| SpamapS | can configure hiding to see if that is it | 19:45 |
| SpamapS | though the tcpdump would also confirm | 19:45 |
| dprince | lifeless: ^^, totally agree about finding the issue. We can do that in parallel. But having a working cloud in the meaning (if we use arp_ignore) seems like a sane work around for now to me | 19:46 |
| lifeless | dprince: as long as we keep one machine setup to provoke the problem :) | 19:47 |
| lifeless | dprince: or two perhaps. | 19:47 |
| dprince | lifeless: this is a client side fix. | 19:47 |
| lifeless | dprince: ignore_arp prevents arps being *replied* to | 19:47 |
| *** rpodolyaka has quit IRC | 19:47 | |
| lifeless | dprince: if we set it on all the machines, presuming that one of them is causing it, we may make the symptom go away | 19:48 |
| lifeless | dprince: and thus not know the cause | 19:48 |
| dprince | lifeless: I've hit this on every machine I've tried | 19:48 |
| dprince | lifeless: any DHCP client machine hits the issue | 19:48 |
| lifeless | dprince: right, so my suggestion is to keep one unaltered ;) | 19:48 |
| lifeless | dprince: do you have em2 ifup'd ? | 19:49 |
| *** rpodolyaka has joined #tripleo | 19:49 | |
| lifeless | dprince: or is it down? | 19:49 |
| dprince | lifeless: easy to do | 19:49 |
| dprince | lifeless: em2 is connected (cabled up) but down (no DHCP lease) | 19:49 |
| dprince | lifeless: on some nodes however we are assigning it a public IP too | 19:49 |
| dprince | lifeless: only in the overcloud though | 19:49 |
| lifeless | dprince: ok, so nodes with em2 *down* are displaying this problem ? I think that rules out arp flux | 19:51 |
| lifeless | anyhow, sure, no objection to experiments in parallel, we just need to keep plugging until we have a clear cause-and-effect model | 19:52 |
| *** nijaba has joined #tripleo | 19:52 | |
| dprince | lifeless: http://paste.openstack.org/show/73524/ | 19:53 |
| *** hashar has quit IRC | 19:53 | |
| dprince | lifeless: tcpdump of em1 on client right before the DHCP server logs the DHCPDECLINE | 19:53 |
| dprince | lifeless: client IP was 10.1.8.75 as obtain by the initial DHCP lease. The renew fails though. | 19:54 |
| *** jistr has quit IRC | 20:05 | |
| SpamapS | dprince: you need -e | 20:08 |
| *** rpodolyaka has quit IRC | 20:10 | |
| dprince | SpamapS: maybe, I can try it again.... | 20:13 |
| *** rcarrillocruz1 has joined #tripleo | 20:13 | |
| *** rcarrillocruz has quit IRC | 20:14 | |
| *** rpodolyaka has joined #tripleo | 20:15 | |
| *** rpodolyaka has quit IRC | 20:16 | |
| *** rpodolyaka has joined #tripleo | 20:18 | |
| *** akuznetsov has quit IRC | 20:20 | |
| *** marun has joined #tripleo | 20:25 | |
| lifeless | DHCPDECLINE happens anytime the dhclient helper scripts fail too | 20:25 |
| lifeless | e.g. | 20:25 |
| lifeless | http://web.archiveorange.com/archive/v/quIQBm01lnJXDhkgcClC | 20:25 |
| *** rwsu has joined #tripleo | 20:28 | |
| *** jtomasek has quit IRC | 20:31 | |
| lifeless | dprince: the time match with lease expiry makes me think its either self interference, or failure to update some metadata locally triggering an error back to dhclient | 20:33 |
| *** fandi has quit IRC | 20:41 | |
| *** julim has quit IRC | 20:47 | |
| *** rpodolyaka has quit IRC | 20:48 | |
| *** jprovazn_afk has quit IRC | 20:49 | |
| *** vkozhukalov_ has quit IRC | 20:53 | |
| *** hashar has joined #tripleo | 20:53 | |
| *** lazy_prince has quit IRC | 20:57 | |
| *** jp_at_hp has joined #tripleo | 21:00 | |
| *** lazy_prince has joined #tripleo | 21:04 | |
| *** lazy_prince is now known as killer_prince | 21:04 | |
| *** lblanchard has quit IRC | 21:04 | |
| *** ohadlevy has joined #tripleo | 21:05 | |
| openstackgerrit | Chris Krelle proposed a change to openstack/tripleo-image-elements: Change nova scheduler driver location https://review.openstack.org/80699 | 21:06 |
| *** sdake_ has quit IRC | 21:10 | |
| *** sdake_ has joined #tripleo | 21:10 | |
| *** sdake_ has quit IRC | 21:10 | |
| *** sdake_ has joined #tripleo | 21:10 | |
| openstackgerrit | A change was merged to openstack/tripleo-heat-templates: Add Wait Condition to Block Storage Templates https://review.openstack.org/80347 | 21:13 |
| *** matty_dubs is now known as matty_dubs|gone | 21:13 | |
| *** openstackgerrit has quit IRC | 21:21 | |
| *** openstackgerrit has joined #tripleo | 21:21 | |
| *** morazi has quit IRC | 21:23 | |
| openstackgerrit | Jon-Paul Sullivan proposed a change to openstack/tripleo-image-elements: Create pip manifest files of installed packages https://review.openstack.org/80644 | 21:24 |
| *** jang1 has joined #tripleo | 21:26 | |
| *** morazi has joined #tripleo | 21:28 | |
| *** rbrady is now known as rbrady-afk | 21:31 | |
| *** openstackgerrit has quit IRC | 21:32 | |
| *** openstackgerrit has joined #tripleo | 21:32 | |
| *** vkozhukalov_ has joined #tripleo | 21:37 | |
| *** rpodolyaka has joined #tripleo | 21:47 | |
| *** jp_at_hp has quit IRC | 21:48 | |
| *** morazi has quit IRC | 21:49 | |
| *** newell has quit IRC | 21:56 | |
| *** rcarrillocruz has joined #tripleo | 22:00 | |
| *** rcarrillocruz1 has quit IRC | 22:02 | |
| *** tburnes has quit IRC | 22:02 | |
| *** d0ugal has joined #tripleo | 22:06 | |
| *** d0ugal has joined #tripleo | 22:06 | |
| Lotus907efi | does easy_install use the PYPI_MIRROR_URL env setting? | 22:07 |
| Lotus907efi | i.e. if I set PYPI_MIRROR_URL to point to a mirror I have setup here will easy_install try to use that mirror? | 22:08 |
| greghaynes | I dont believe so | 22:09 |
| Lotus907efi | crud | 22:09 |
| greghaynes | I think there was talk about that yesterday *looks at scrollback* | 22:11 |
| Lotus907efi | the devtest.sh run gets to a point where the pbr python module is being pulled in by a dependency and it fails to get the module | 22:11 |
| *** bauzas has joined #tripleo | 22:11 | |
| Lotus907efi | even though previous modules not pulled in through easy_install worked | 22:11 |
| Lotus907efi | and the pbr module is there in my mirror | 22:11 |
| Lotus907efi | well I thought setting up a pypi mirror internal to our DC would solve the problem | 22:12 |
| Lotus907efi | but it appears this is still a stumbling block | 22:12 |
| Lotus907efi | crud | 22:12 |
| *** jrist has quit IRC | 22:20 | |
| *** jrist has joined #tripleo | 22:20 | |
| jang1 | Hey Lotus907efi. | 22:20 |
| Lotus907efi | hey | 22:21 |
| jang1 | Seeing similar things today. I'm getting rapidly tired of trying to wrap easy_install in layers of proxy. Think it might be better to tackle the problem at source. | 22:22 |
| Lotus907efi | ha! | 22:22 |
| lifeless | Lotus907efi: the pypi element sets ~..pydistutils.cfg | 22:24 |
| lifeless | Lotus907efi: which easy_install uses | 22:24 |
| lifeless | Lotus907efi: note that the install of client tools for the incubator (outside built elements) isn't affected by the pypi element - you should configure your user mirror directly. | 22:26 |
| lifeless | Lotus907efi: e.g. in ~/.pip/pip.conf and ~/.pydistutils.cfg | 22:26 |
| *** rpodolyaka has quit IRC | 22:26 | |
| lifeless | Lotus907efi: see the pypi element pre-install hook for examples | 22:26 |
| Lotus907efi | you mean like kill the code with a big eraser? | 22:26 |
| Lotus907efi | I thought I heard mordred mention something about how easy_install is not something that we have the ability to alter | 22:26 |
| Lotus907efi | ok | 22:26 |
| Lotus907efi | yeah, I found that | 22:26 |
| Lotus907efi | but I don't have that file here | 22:26 |
| Lotus907efi | but I guess I could create it | 22:26 |
| Lotus907efi | can I put a redirection URL in that file for easy_install to use? | 22:26 |
| Lotus907efi | hmmm | 22:26 |
| Lotus907efi | ok | 22:26 |
| Lotus907efi | I found some docs talking about this earlier today so I will have to figure out where I found them | 22:26 |
| greghaynes | Maybe that was http_proxy - thats what I was remembering as well | 22:27 |
| Lotus907efi | I seem to recall reading something today that talked about the .pydistutils.cfg file and that was why I looked for it here in my env | 22:28 |
| Lotus907efi | so by setting find_links to point to my pypi mirror in .pydistutils.cfg, easy_install will try to get stuff there? | 22:31 |
| mordred | the easy_install issue is not getting http_proxy passed through to it when it's invoked to satisfy setup_requires | 22:40 |
| mordred | adding a local pypi mirror into .pydistutils.cfg should work just fine | 22:41 |
| *** marun has quit IRC | 22:42 | |
| Lotus907efi | ok, that is what I am trying now | 22:42 |
| Lotus907efi | i.e. adding a local pypi mirror into .pydistutils.cfg | 22:42 |
| Lotus907efi | starting devtest.sh again | 22:43 |
| Lotus907efi | and crossing my fingers & toes | 22:43 |
| *** hashar has quit IRC | 22:44 | |
| *** rpodolyaka has joined #tripleo | 22:49 | |
| Lotus907efi | nope, didn't work | 22:49 |
| Lotus907efi | still failed to find pbr at the same spot | 22:49 |
| Lotus907efi | and the apache access logs stop right after fetching os-refresh-config-0.1.1.tar.gz | 22:50 |
| Lotus907efi | so it never gets a request for the pbr stuff'o | 22:50 |
| Lotus907efi | in my ~/.pydistutils.cfg file I have: | 22:51 |
| Lotus907efi | [easy_install] | 22:51 |
| Lotus907efi | index_url = http://15.126.28.132/pypi/mirror/ | 22:51 |
| Lotus907efi | I am running devtest.sh as root | 22:52 |
| jang1 | where's i breaking, Lotus907efi? | 22:52 |
| jang1 | yup, but is it stil doing (say) openstack-tools installation, or building the seed? | 22:52 |
| Lotus907efi | and the .pydistutils.cfg is in root's home dir | 22:52 |
| Lotus907efi | ummm, I am not sure.... let me look through the logs and see where it is exactly | 22:53 |
| jang1 | you can try ~/.cache/tripleo/tripleo-incubator/openstack-tools/bin/pip freeze | 22:54 |
| lifeless | Lotus907efi: I think you need to pastebin the whole log | 22:54 |
| jang1 | and see if it looks like that's full of python-*clients or not | 22:54 |
| lifeless | Lotus907efi: there are multiple places it may be failing, and its not clear where you are changing things/altering settings | 22:54 |
| Lotus907efi | ok | 22:54 |
| lifeless | (I'm not here - its saturday) | 22:54 |
| jang1 | the PYPI_MIRROR_URL is not honoured by the setup-clienttools or whatever that script is called. | 22:55 |
| jang1 | I've a small patch for that. | 22:55 |
| jang1 | (you don't notice that if you're also running a squid proxy, I think) | 22:56 |
| jang1 | it's been very useful having a bunch of developers here running the setup for the first time. | 22:56 |
| jang1 | there are many things that you can just work around once, then they stay fixed; I'm trying to get to the point where I've actually got fixes for all of those. | 22:56 |
| jang1 | first impressions, and all that | 22:56 |
| *** jang1 has quit IRC | 23:01 | |
| Lotus907efi | http://pastebin.com/AZsnNLVi | 23:06 |
| Lotus907efi | complete log from this devtest.sh run | 23:07 |
| Lotus907efi | what is "~/.cache/tripleo/tripleo-incubator/openstack-tools/bin/pip freeze" going to do exactly? | 23:08 |
| SpamapS | Lotus907efi: that just prints out a requirements.txt file of what is installed now in the venv | 23:12 |
| * SpamapS -> kids, EOD, etc | 23:12 | |
| Lotus907efi | ah, ok | 23:18 |
| Lotus907efi | there are about eight python-blah things listed there | 23:19 |
| *** e0ne has joined #tripleo | 23:21 | |
| *** e0ne_ has quit IRC | 23:22 | |
| *** Lotus907efi is now known as Lotus907efi- | 23:27 | |
| *** e0ne has quit IRC | 23:30 | |
| *** e0ne has joined #tripleo | 23:30 | |
| *** rpodolyaka has quit IRC | 23:32 | |
| *** openstackstatus has quit IRC | 23:36 | |
| *** Lotus907efi- has quit IRC | 23:36 | |
| *** e0ne has quit IRC | 23:37 | |
| *** rwsu has quit IRC | 23:38 | |
| *** rbrady-afk has quit IRC | 23:38 | |
| *** openstackgerrit has quit IRC | 23:39 | |
| *** killer_prince has quit IRC | 23:42 | |
| *** nati_ueno has quit IRC | 23:42 | |
| *** sdake has quit IRC | 23:43 | |
| *** rcarrillocruz has quit IRC | 23:46 | |
| *** cwolferh_ has quit IRC | 23:46 | |
| *** SpamapS has quit IRC | 23:46 | |
| *** bogdando has quit IRC | 23:46 | |
| *** cody-somerville has quit IRC | 23:46 | |
| *** tomh_ has quit IRC | 23:46 | |
| *** rollerj__ has quit IRC | 23:46 | |
| *** funzo has quit IRC | 23:46 | |
| *** sseago_ has quit IRC | 23:46 | |
| *** Hefeweizen has quit IRC | 23:46 | |
| *** sdague has quit IRC | 23:46 | |
| *** jb11211 has quit IRC | 23:46 | |
| *** rainya has quit IRC | 23:46 | |
| *** echohead has quit IRC | 23:46 | |
| *** mkerrin1 has quit IRC | 23:46 | |
| *** andrearosa has quit IRC | 23:46 | |
| *** cody-somerville- has quit IRC | 23:46 | |
| *** Daviey has quit IRC | 23:46 | |
| *** dtantsur has quit IRC | 23:46 | |
| *** jrist has quit IRC | 23:46 | |
| *** d0ugal has quit IRC | 23:46 | |
| *** vkozhukalov_ has quit IRC | 23:46 | |
| *** EntropyWorks has quit IRC | 23:46 | |
| *** Matt2 has quit IRC | 23:46 | |
| *** antonym has quit IRC | 23:46 | |
| *** dguerri has quit IRC | 23:46 | |
| *** smulcahy has quit IRC | 23:46 | |
| *** shortstop has quit IRC | 23:46 | |
| *** bauzas has quit IRC | 23:46 | |
| *** weshay has quit IRC | 23:46 | |
| *** victor_lowther_ has quit IRC | 23:46 | |
| *** lsmola has quit IRC | 23:46 | |
| *** AaronGr has quit IRC | 23:46 | |
| *** jeremydei has quit IRC | 23:46 | |
| *** dividebin has quit IRC | 23:46 | |
| *** tzumainn has quit IRC | 23:46 | |
| *** dprince has quit IRC | 23:46 | |
| *** blamar has quit IRC | 23:46 | |
| *** ccorrigan has quit IRC | 23:46 | |
| *** lynxman has quit IRC | 23:46 | |
| *** vipul has quit IRC | 23:46 | |
| *** sbadia has quit IRC | 23:46 | |
| *** dkehn has quit IRC | 23:46 | |
| *** ewindisch has quit IRC | 23:46 | |
| *** Shrews has quit IRC | 23:46 | |
| *** clarkb has quit IRC | 23:46 | |
| *** kevinbenton has quit IRC | 23:46 | |
| *** ohadlevy has quit IRC | 23:46 | |
| *** ccrouch has quit IRC | 23:46 | |
| *** mikal has quit IRC | 23:46 | |
| *** juice has quit IRC | 23:46 | |
| *** davidlenwell has quit IRC | 23:46 | |
| *** Slower has quit IRC | 23:46 | |
| *** yamahata_ has quit IRC | 23:46 | |
| *** rha has quit IRC | 23:46 | |
| *** nijaba has quit IRC | 23:46 | |
| *** ifarkas has quit IRC | 23:46 | |
| *** slagle has quit IRC | 23:46 | |
| *** jang has quit IRC | 23:46 | |
| *** ProfFalken has quit IRC | 23:46 | |
| *** matty_dubs|gone has quit IRC | 23:46 | |
| *** dkehn_ has quit IRC | 23:46 | |
| *** sgrasley has quit IRC | 23:46 | |
| *** jeblair has quit IRC | 23:46 | |
| *** jpeeler has quit IRC | 23:46 | |
| *** sff has quit IRC | 23:46 | |
| *** SloppyJ_ has quit IRC | 23:46 | |
| *** michchap has quit IRC | 23:46 | |
| *** DuncanT- has quit IRC | 23:46 | |
| *** pleia2 has quit IRC | 23:46 | |
| *** shadower has quit IRC | 23:46 | |
| *** spiffxp_ has quit IRC | 23:46 | |
| *** jcooley has quit IRC | 23:46 | |
| *** flashgordon has quit IRC | 23:46 | |
| *** gilliard has quit IRC | 23:46 | |
| *** lifeless has quit IRC | 23:46 | |
| *** jesusaurus has quit IRC | 23:46 | |
| *** marios has quit IRC | 23:46 | |
| *** mordred has quit IRC | 23:46 | |
| *** lucasagomes has quit IRC | 23:46 | |
| *** al has quit IRC | 23:46 | |
| *** zigo has quit IRC | 23:46 | |
| *** tteggel has quit IRC | 23:46 | |
| *** StevenK has quit IRC | 23:46 | |
| *** GheRivero has quit IRC | 23:46 | |
| *** openstack has joined #tripleo | 23:47 | |
| *** rpodolyaka1 has joined #tripleo | 23:47 | |
| *** openstackstatus has joined #tripleo | 23:47 | |
| *** Lotus907efi1 has joined #tripleo | 23:48 | |
| *** echohead has joined #tripleo | 23:51 | |
| *** mkerrin has joined #tripleo | 23:51 | |
| *** cody-somerville- has joined #tripleo | 23:56 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!