| *** e0ne has joined #tripleo | 00:01 | |
| SpamapS | lifeless: mmk. I'm available for any parallel tasks you might have. | 00:01 |
|---|---|---|
| *** ci-overcloud has joined #tripleo | 00:03 | |
| ci-overcloud | ************** ci-overcloud complete status=6 ************ | 00:03 |
| *** ci-overcloud has quit IRC | 00:03 | |
| *** e0ne_ has joined #tripleo | 00:03 | |
| *** e0ne has quit IRC | 00:03 | |
| lifeless | SpamapS: more automation of the stuff in line 235 of https://etherpad.openstack.org/p/tripleo-test-cluster or fixing the critical bugs related to this like the dhcp-all-interfaces breaking-us would be good | 00:04 |
| SpamapS | lifeless: what does dhcp-all-interfaces break? | 00:05 |
| lifeless | ok FINALLY got it past all the networking and its now building off local pypi and ubuntu mirrors | 00:05 |
| lifeless | SpamapS: https://trello.com/c/ruXSHgYq/85-dhcp-all-interfaces-breaking-our-brains | 00:05 |
| lifeless | running up the street for lynne, bbs, hopefully this completes ok | 00:06 |
| StevenK | I am also running up the street. | 00:06 |
| StevenK | Our postie here when confronted with a parcel will either try and force it into the mailbox, or give up and put up a card in the mailbox. | 00:06 |
| StevenK | So now I have to walk up to the post office. | 00:07 |
| *** e0ne_ has quit IRC | 00:08 | |
| *** rlandy has quit IRC | 00:08 | |
| tchaypo | except not right now, because it won't be at the post office until he finshes his rounds and takes it back | 00:08 |
| *** ci-overcloud has joined #tripleo | 00:10 | |
| ci-overcloud | ************** ci-overcloud complete status=1 ************ | 00:10 |
| *** ci-overcloud has quit IRC | 00:10 | |
| *** yamahata has quit IRC | 00:23 | |
| *** matsuhashi has joined #tripleo | 00:23 | |
| lifeless | SpamapS: 2014-03-01 09:27:02.030 4726 WARNING heat.openstack.common.db.sqlalchemy.session [-] This application has not enabled MySQL traditional mode, which means silent data corruption may occur. Please encourage the application developers to enable this mode. | 00:36 |
| lifeless | ERROR: (IntegrityError) (1451, 'Cannot delete or update a parent row: a foreign key constraint fails (`heat`.`stack_lock`, CONSTRAINT `stack_lock_ibfk_1` FOREIGN KEY (`stack_id`) REFERENCES `stack` (`id`))') 'DELETE FROM stack WHERE stack.id = %s' ('1f485871-5c05-4118-aedb-14aa8f8d6962',) | 00:36 |
| lifeless | heat-manage purge_deleted is spamming cron | 00:36 |
| StevenK | tchaypo: Tis, since the card was put in my mailbox on Friday | 00:41 |
| StevenK | Although the postie here is *useless*. | 00:42 |
| StevenK | He wrote the street number on the card, but not the street name, so I got asked 20 questions by the guy at the post office. | 00:42 |
| SpamapS | lifeless: right that is a bug that was fixed | 00:43 |
| SpamapS | lifeless: so newer Heat should take care of it. | 00:43 |
| StevenK | "This application has not enabled MySQL traditional mode," -- why must this be logged at all? Srsly :-( | 00:44 |
| SpamapS | StevenK: reasons | 00:44 |
| SpamapS | StevenK: because nobody believed in Drizzle basically ;) | 00:45 |
| * StevenK tries to work out how to open this parcel without resorting to power tools. | 00:45 | |
| *** ci-overcloud has joined #tripleo | 00:48 | |
| ci-overcloud | ************** ci-overcloud complete status=6 ************ | 00:48 |
| *** ci-overcloud has quit IRC | 00:48 | |
| StevenK | lifeless: I'm mostly afk, installing more RAM in my desktop. | 00:49 |
| lifeless | glwt | 00:52 |
| lifeless | user image build fail. stabbbbity | 00:53 |
| lifeless | I probably should mention how MUCH SO VERY VERY MUCH I want our HA working right now | 01:00 |
| *** e0ne has joined #tripleo | 01:01 | |
| *** funzo_ has joined #tripleo | 01:04 | |
| *** spzala has joined #tripleo | 01:05 | |
| *** jeblair_ has joined #tripleo | 01:08 | |
| *** matty_du1s|gone has joined #tripleo | 01:09 | |
| *** Hefeweiz1n has joined #tripleo | 01:09 | |
| *** kiall_ has joined #tripleo | 01:10 | |
| *** cody-somerville has quit IRC | 01:13 | |
| *** Matt2 has quit IRC | 01:13 | |
| *** decede has quit IRC | 01:13 | |
| *** ewindisch has quit IRC | 01:13 | |
| *** jeblair has quit IRC | 01:13 | |
| *** Daviey has quit IRC | 01:13 | |
| *** e0ne has quit IRC | 01:13 | |
| *** killer_prince has quit IRC | 01:13 | |
| *** kiall has quit IRC | 01:13 | |
| *** funzo has quit IRC | 01:13 | |
| *** rbrady has quit IRC | 01:13 | |
| *** sseago has quit IRC | 01:13 | |
| *** matty_dubs|gone has quit IRC | 01:13 | |
| *** Hefeweizen has quit IRC | 01:13 | |
| *** Daviey has joined #tripleo | 01:14 | |
| StevenK | steven@undermined:~% free -m | head -n 2 | tail -n 1 | 01:16 |
| StevenK | Mem: 24007 1056 22951 0 89 536 | 01:16 |
| StevenK | Heh. Used is just over 1GiB. Now let's start Firefox | 01:16 |
| lifeless | 24GB? | 01:18 |
| StevenK | lifeless: Already had 8, bought 16 | 01:19 |
| *** ewindisch has joined #tripleo | 01:19 | |
| *** decede has joined #tripleo | 01:19 | |
| mordred | SpamapS: we tried to get people to switch to drizzle... | 01:19 |
| *** cody-somerville has joined #tripleo | 01:20 | |
| StevenK | lifeless: You were mentioning >16G on the list, and the 8G I already had is quite fast Ripjaws RAM, so I didn't want to lose it | 01:20 |
| *** rbrady has joined #tripleo | 01:21 | |
| *** yamahata has joined #tripleo | 01:21 | |
| *** killer_prince has joined #tripleo | 01:21 | |
| StevenK | mordred: And people stuck to MySQL anyway? | 01:21 |
| StevenK | Or MariaDB | 01:21 |
| *** Matt2 has joined #tripleo | 01:22 | |
| *** matsuhashi has quit IRC | 01:24 | |
| lifeless | StevenK: cool | 01:24 |
| *** matsuhashi has joined #tripleo | 01:26 | |
| mordred | StevenK: well, yeah | 01:29 |
| mordred | StevenK: the reason people didn't move to drizzle is that it would have involved moving to something away from what they were already using. it wasn't that drizzle wasn't better - it was that change incurs cost | 01:30 |
| mordred | StevenK: so they're pretty unlikely to change to a different non-mysql database either | 01:31 |
| *** nosnos has joined #tripleo | 01:31 | |
| mordred | unless they go in a completely different direction and do something like mongo | 01:31 |
| *** sseago_ has joined #tripleo | 01:31 | |
| mordred | StevenK: nice new ram, btw | 01:33 |
| StevenK | Now my desktop can stop swapping to death for ~15 minutes whenever I spin up TripleO | 01:34 |
| *** noslzzp has quit IRC | 01:35 | |
| mordred | w00t | 01:35 |
| StevenK | mordred: Yeah. I know the decision to fork off wasn't taken lightly, but it is a cost that is constantly paid. | 01:35 |
| mordred | yup | 01:35 |
| StevenK | mordred: Did Drizzle make that a default, or something else, getting back to the log message that started this discussion. | 01:36 |
| mordred | StevenK: yeah - we got rid of sql modes all together and just made evrything strict | 01:37 |
| StevenK | And how did that affect speed? | 01:37 |
| mordred | not at all | 01:37 |
| StevenK | You don't wait for fsync() and friends on COMMIT ? | 01:38 |
| mordred | that's not related to sql mode | 01:38 |
| StevenK | Ah | 01:38 |
| mordred | the option you're thinking of is related to the replication binary log | 01:39 |
| *** noslzzp has joined #tripleo | 01:40 | |
| mordred | commit on innodb tables is always crash safe - the question in tuning is whether or not you care if you lose something from the binlog on crash | 01:41 |
| mordred | that was one of the things we solved in drizzle by having the replication log actualy written into a table inside of innodb, so that it coudl participate in the commit machinery and not need any 2pc semantics between db files and replication logs | 01:42 |
| mordred | StevenK: "This application has not enabled MySQL traditional mode," is more about incoming sql processing | 01:45 |
| mordred | without turning on traditional or strict mode, mysql will endeavor to do _something_ with the data you give it, even if that violates data typing | 01:46 |
| mordred | so you can get truncation of values, or weird values inserted into columns you didn't expect | 01:46 |
| mordred | this was considered a "feature" back in the very early web days | 01:47 |
| mordred | but is now widely regarded as a very bad idea | 01:47 |
| StevenK | And made a lot of people very angry? :-P | 01:48 |
| mordred | :) | 01:48 |
| mordred | it's also an actually important thing if you have any non-transactional tables, such as myisam | 01:48 |
| mordred | because throwing an error on insert in the middle of a multi-statement sequence, without rollback, could actually be much worse :) | 01:49 |
| mordred | in any case, we should probbly be setting sql_mode=STRICT_TRANS_TABLES at the very least | 01:49 |
| * mordred will look in to how we can tell sqlalchmey to do that, if possible | 01:50 | |
| mordred | SpamapS: any thoughts from you on current state of thinking regarding sql_mode and openstack? | 01:53 |
| *** sdake has quit IRC | 01:54 | |
| *** Daviey has quit IRC | 02:00 | |
| *** morazi has quit IRC | 02:00 | |
| lifeless | yay all machines active | 02:01 |
| *** ci-overcloud has joined #tripleo | 02:01 | |
| ci-overcloud | ************** ci-overcloud complete status=1 ************ | 02:01 |
| *** ci-overcloud has quit IRC | 02:01 | |
| lifeless | now to wait and see dammit | 02:01 |
| lifeless | + wait_for_stack_ready 300 10 ci-overcloud | 02:01 |
| lifeless | Command output matched 'CREATE_FAILED'. Exiting... | 02:01 |
| *** e0ne has joined #tripleo | 02:01 | |
| lifeless | | stack_status_reason | Resource create failed: WaitConditionTimeout: 0 of 1 | | 02:01 |
| lifeless | SpamapS: how does one make heat wait longer ? | 02:02 |
| lifeless | nvm | 02:02 |
| * lifeless tries again | 02:03 | |
| *** e0ne has quit IRC | 02:06 | |
| *** spzala has quit IRC | 02:13 | |
| *** eghobo has quit IRC | 02:19 | |
| *** ramishra has joined #tripleo | 02:21 | |
| *** xuhaiwei has joined #tripleo | 02:33 | |
| *** rbrady has quit IRC | 02:37 | |
| *** rlandy has joined #tripleo | 02:37 | |
| *** rlandy has quit IRC | 02:37 | |
| xuhaiwei | lifeless: I haven't solved this problem "Could not fetch URL https://pypi.python.org/simple/pip/: timed out" yet, so how to use the mirror, should I create it by myself? | 02:42 |
| lifeless | xuhaiwei: yes, you'll need to create it yourself | 02:42 |
| lifeless | there is some documentation about that in diskimage-builder/elements/pypi/README.md | 02:43 |
| xuhaiwei | ok, thanks | 02:43 |
| *** sdake has joined #tripleo | 02:45 | |
| lifeless | dib-run-parts Mon Mar 31 02:51:37 UTC 2014 Running /opt/stack/os-config-refresh/post-configure.d/80-neutron-openvswitch-agent | 02:46 |
| lifeless | + os-svc-enable -n neutron-openvswitch-agent | 02:47 |
| lifeless | + os-svc-restart -n neutron-openvswitch-agent | 02:47 |
| lifeless | stop: Unknown instance: | 02:47 |
| lifeless | start: Job failed to start | 02:47 |
| lifeless | 2014-03-31 02:51:40.029 9407 ERROR neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-012f586a-c785-45be-b77a-6213cb13f969 None] Bridge br-untagged for physical network tripleo-bm-test does not exist. Agent terminated! | 02:47 |
| lifeless | oh for the log of dog | 02:53 |
| lifeless | /opt/stack/tripleo-incubator/scripts/devtest_overcloud.sh: line 237: jq: command not found | 02:53 |
| lifeless | undercloud is quantal | 02:53 |
| StevenK | lifeless: I *think* -backports is safe enough for one package | 02:56 |
| *** untriaged-bot has joined #tripleo | 03:00 | |
| untriaged-bot | Untriaged bugs so far: | 03:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1290488 | 03:00 |
| uvirtbot | Launchpad bug 1290488 in tripleo "Baremetal: Invalid credentials" [Undecided,Incomplete] | 03:00 |
| *** untriaged-bot has quit IRC | 03:00 | |
| *** e0ne has joined #tripleo | 03:01 | |
| *** e0ne has quit IRC | 03:03 | |
| *** eghobo has joined #tripleo | 03:07 | |
| *** matsuhashi has quit IRC | 03:19 | |
| cody-somerville | How much disk space is required for full mirror of all things tripleo devtest needs? | 03:21 |
| StevenK | cody-somerville: ~150-180G or so? | 03:25 |
| lifeless | cody-somerville: saucy (trusty soon) + 100MB or so from pypi (we can partial mrror that, unlike Ubuntu) | 03:30 |
| lifeless | cody-somerville: + a base os image | 03:30 |
| lifeless | cody-somerville: + git trees | 03:30 |
| *** xuhaiwei has quit IRC | 03:31 | |
| StevenK | Ubuntu is ~30G per distroarchseries or so | 03:33 |
| StevenK | With source being effectively another DAS | 03:33 |
| *** nosnos has quit IRC | 03:34 | |
| * StevenK increases the memory pressue his desktop is under and starts Thunderbird. | 03:39 | |
| StevenK | Oh noes, I'm into swap | 03:39 |
| *** matsuhashi has joined #tripleo | 03:57 | |
| *** e0ne has joined #tripleo | 04:01 | |
| *** nosnos has joined #tripleo | 04:02 | |
| lifeless | StevenK: heh | 04:03 |
| *** e0ne has quit IRC | 04:06 | |
| *** xuhaiwei has joined #tripleo | 04:06 | |
| StevenK | So with a devtest running 4 nodes + seed, firefox and thunderbird, I'm stable at Mem: 24007 23192 814 0 27 13240 and ~45M into swap | 04:11 |
| lifeless | swapoff -a | 04:12 |
| StevenK | Does that force the pages back into RAM? | 04:12 |
| lifeless | yeah | 04:12 |
| StevenK | I think I can cope with 45M swap usage, TBH | 04:13 |
| StevenK | Bleh | 04:13 |
| StevenK | Error saving domain: internal error: unable to execute QEMU command 'migrate': State blocked by non-migratable device '0000:00:03.0/ich9_ahci' | 04:13 |
| lifeless | Running setup.py (path:/opt/stack/venvs/python-glanceclient/build/cryptography/setup.py) egg_info for package cryptography | 04:13 |
| lifeless | No local packages or download links found for six>=1.4.1 | 04:13 |
| lifeless | switch | 04:14 |
| lifeless | http://mirror.robertcollins.net/pypi/simple/six/ has plenty newer. wtf | 04:14 |
| greghaynes | I think six was another one of the easy-installed dependencies | 04:14 |
| greghaynes | Ran into that when I had a wheel only pypi mirror | 04:14 |
| StevenK | Host mirror.robertcollins.net not found: 3(NXDOMAIN) | 04:14 |
| StevenK | :-P | 04:15 |
| lifeless | File "/opt/stack/venvs/python-glanceclient/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 587, in easy_install | 04:15 |
| lifeless | raise DistutilsError(msg) | 04:15 |
| StevenK | I was pondering setting up a pypi mirror | 04:15 |
| StevenK | What's the daily churn? | 04:15 |
| greghaynes | StevenK: no idea, I just stay a few days out of date :p | 04:15 |
| *** ramishra_ has joined #tripleo | 04:16 | |
| StevenK | greghaynes: Because WCPGW? :-P | 04:16 |
| lifeless | StevenK: my bill hasn't noticed yet | 04:16 |
| lifeless | StevenK: I'm running bandersnatch, and then pypi-mirror on top of that to get wheels for openstack stuff | 04:16 |
| greghaynes | wow, double mirror | 04:16 |
| *** ramishra has quit IRC | 04:17 | |
| * StevenK kills nodes, since migration save isn't working, and teh googles says "New qemu lol" | 04:17 | |
| lifeless | greghaynes: whats the meme | 04:17 |
| lifeless | double rainbow | 04:17 |
| greghaynes | double mirror all the way across the sky | 04:17 |
| StevenK | Hm, I don't know that one | 04:18 |
| greghaynes | oh my | 04:18 |
| lifeless | righto fixing the order of mirror urls fixed that | 04:18 |
| lifeless | easy f*ing install | 04:18 |
| StevenK | lifeless: Yes, I find easy install is a double lie | 04:18 |
| lifeless | I bet Lotus907efi will hit that tomorrow | 04:18 |
| greghaynes | Awesome, so they were in the wrong order after all | 04:18 |
| lifeless | greghaynes: there was never any doubt | 04:18 |
| greghaynes | I think I mentioned they were backwards to him or someone in hipchat when we were debugging something | 04:19 |
| greghaynes | hopefully it was him, I can never keep the name mappings together in my head | 04:19 |
| lifeless | 6/10 hypervisors have checked in | 04:23 |
| lifeless | sheese | 04:23 |
| *** nati_ueno has joined #tripleo | 04:23 | |
| lifeless | slowest deploy evah | 04:25 |
| lifeless | yay 7 | 04:25 |
| StevenK | lifeless: Let me take the new RAM out of my desktop, and I'd question that. | 04:26 |
| StevenK | TripleO on 8G results in a LOT of thrashing | 04:26 |
| lifeless | StevenK: 35m so far | 04:26 |
| lifeless | StevenK: yeah, don't do that. | 04:26 |
| StevenK | Haha | 04:26 |
| lifeless | StevenK: so where are we at with thingy | 04:27 |
| StevenK | Thingy? | 04:27 |
| lifeless | StevenK: register-nodes etc stuff | 04:27 |
| *** marun has joined #tripleo | 04:27 | |
| *** marun is now known as marun_afk | 04:27 | |
| StevenK | lifeless: I'm going to guess you mean the movement to os-cloud-config, while I try and swap context back in. | 04:28 |
| lifeless | StevenK: that and making more flavours to match the hardware supplied | 04:28 |
| lifeless | StevenK: and being idempotent | 04:29 |
| StevenK | lifeless: So you mentioned this at the sprint, and then I got distracted by moving the seed into the JSON | 04:29 |
| openstackgerrit | Steve Kowalik proposed a change to openstack/tripleo-incubator: Store seed details in the JSON https://review.openstack.org/79051 | 04:32 |
| lifeless | one more hypervisor to check in | 04:33 |
| *** lazy_prince has joined #tripleo | 04:39 | |
| *** matsuhashi has quit IRC | 04:39 | |
| *** marun_afk has quit IRC | 04:39 | |
| *** sseago_ has quit IRC | 04:39 | |
| *** sdake has quit IRC | 04:39 | |
| *** yamahata has quit IRC | 04:39 | |
| *** killer_prince has quit IRC | 04:39 | |
| *** lazy_prince is now known as killer_prince | 04:39 | |
| *** matsuhashi has joined #tripleo | 04:43 | |
| *** sdake has joined #tripleo | 04:45 | |
| *** marun has joined #tripleo | 04:46 | |
| *** yamahata has joined #tripleo | 04:46 | |
| *** sseago_ has joined #tripleo | 04:47 | |
| lifeless | Unable to establish connection to http://138.35.77.16:35357/v2.0/endpoints | 04:47 |
| lifeless | -stab- | 04:47 |
| StevenK | lifeless: So, mkdir scripts, cp ../tripleo-incubator/scripts/register-nodes scripts and convert to Python? | 04:51 |
| lifeless | at the risk of wedging the undercloud, I'm going to bounce the mellanox driver | 04:51 |
| lifeless | StevenK: thats too brief for me to eval properly | 04:51 |
| StevenK | lifeless: register-nodes | 04:51 |
| lifeless | yes | 04:51 |
| StevenK | Well, register-nodes -> os-cloud-config | 04:52 |
| lifeless | go on | 04:52 |
| StevenK | Well, scripts is silly | 04:52 |
| StevenK | Oh, we want os_cloud_config/nodes.py or something? | 04:52 |
| openstackgerrit | lifeless proposed a change to openstack/tripleo-incubator: Add more tools. https://review.openstack.org/84049 | 04:52 |
| lifeless | StevenK: the goal is a Python API for tuskar and a CLI for us to use on the seed | 04:53 |
| lifeless | on/against | 04:53 |
| lifeless | there are lots of routes to that nirvana | 04:53 |
| lifeless | uhm | 04:53 |
| lifeless | the code needs to change repository | 04:53 |
| lifeless | we need to use os-cloud-config (^ my review) | 04:54 |
| lifeless | we need to add a python API | 04:54 |
| lifeless | and we have features we want to add that probably imply a rewrite in pythno | 04:54 |
| lifeless | I'm fairly agnostic about ordering | 04:54 |
| lifeless | but | 04:54 |
| *** sdake has quit IRC | 04:54 | |
| lifeless | do suggest we keep the period of time where we have duplication to a minimium | 04:54 |
| *** akuznetsov has joined #tripleo | 04:55 | |
| lifeless | also eep | 04:57 |
| lifeless | register-nodes --help | 04:57 |
| lifeless | /home/robertc/work/tripleo-incubator/scripts/register-nodes: line 10: $2: unbound variable | 04:57 |
| StevenK | No --help or -h in register-nodes | 04:57 |
| StevenK | It doesn't use getopt | 04:57 |
| lifeless | yeah, thing to fix :) | 04:58 |
| StevenK | ITYM ignore until we rewrite it | 04:59 |
| lifeless | same same | 04:59 |
| lifeless | tomorrow right? | 04:59 |
| StevenK | Hah | 04:59 |
| lifeless | how hard is subprocess.call :) | 05:00 |
| StevenK | We are seriously not suggesting that as the first step? :-( | 05:00 |
| StevenK | I'm sketching out in my head how I want this to look | 05:01 |
| lifeless | I'm typing therefor I'm trolling | 05:01 |
| *** e0ne has joined #tripleo | 05:01 | |
| StevenK | That was well known | 05:02 |
| *** e0ne has quit IRC | 05:04 | |
| lifeless | do I get a well known port of my own? | 05:04 |
| StevenK | Yes. -1 | 05:06 |
| lifeless | LOL | 05:06 |
| StevenK | lifeless: So, os_cloud_config/nodes.py to contain bits to register a single node with nova-bm or ironic as well as some machinery to weed out nodes already registered, and scripts/register-nodes to call into it ? | 05:08 |
| lifeless | StevenK: os_cloud_config/cmds/nodes.py as the CLI entry point | 05:10 |
| lifeless | StevenK: (openstack convention - see nova for instance) | 05:10 |
| lifeless | sorry | 05:10 |
| lifeless | cmd not cmds | 05:11 |
| lifeless | StevenK: I'd make the interface set based - give the CLI a nodes JSON, it parses and passes the vector to os_cloud_config/nodes.py | 05:11 |
| StevenK | lifeless: I was going to keep that logic in the module | 05:15 |
| lifeless | StevenK: up to you | 05:16 |
| lifeless | StevenK: just since tuskar will have the vector as python objects seems like the public pythoninterface doesn't need to know that its 'JSON' | 05:16 |
| StevenK | lifeless: The CLI calls a method in the module, passing the parsed JSON in, and gets back a list of missing nodes | 05:16 |
| StevenK | lifeless: Yes, I was going to keep the JSON contained in the CLI only | 05:17 |
| lifeless | tchaypo: where can we record https://etherpad.openstack.org/p/tripleo-newdev-notes so that its discoverable by new devs | 05:17 |
| lifeless | StevenK: missing nodes? | 05:17 |
| *** noslzzp has quit IRC | 05:18 | |
| StevenK | Missing nodes, IE, that ironic/nova-bm do not know about. | 05:18 |
| StevenK | Then the CLI will register them | 05:18 |
| *** killer_prince has quit IRC | 05:19 | |
| StevenK | lifeless: I don't like the idea of os.environ["USE_IRONIC"] in the CLI either | 05:19 |
| lifeless | StevenK: I think you're setting yourself up for potato programming with that interface | 05:20 |
| lifeless | StevenK: you could see if there is an ironic endpoint registered in keystone and use that to decide | 05:20 |
| *** kevinbenton has left #tripleo | 05:25 | |
| tchaypo | lifeless: good question. The wiki might be a good place for it? | 05:31 |
| tchaypo | lifeless: esp since I'd love to take advantage of some of the workup. Moving it into doc/source/new-dev.rst might be good too | 05:32 |
| *** e0ne has joined #tripleo | 05:32 | |
| *** killer_prince has joined #tripleo | 05:34 | |
| *** e0ne has quit IRC | 05:36 | |
| lifeless | yup, damn mellanox driver | 05:41 |
| *** ci-overcloud has joined #tripleo | 05:43 | |
| ci-overcloud | ************** ci-overcloud complete status=0 ************ | 05:43 |
| *** ci-overcloud has quit IRC | 05:43 | |
| * StevenK stabs setup.py | 05:44 | |
| StevenK | Build the console scripts in place, you stupid tool | 05:44 |
| lifeless | StevenK: hey! | 05:44 |
| lifeless | :> | 05:44 |
| lifeless | StevenK: pip install -e . | 05:45 |
| StevenK | lifeless: Which tries to then write to /usr/local/lib, which is not in-place. | 05:46 |
| lifeless | StevenK: right - mkvirtualenv FTW | 05:47 |
| *** nosnos_ has joined #tripleo | 05:53 | |
| *** nosnos has quit IRC | 05:53 | |
| *** matsuhashi has quit IRC | 05:54 | |
| *** matsuhashi has joined #tripleo | 05:55 | |
| *** e0ne has joined #tripleo | 06:01 | |
| *** e0ne has quit IRC | 06:02 | |
| *** jtomasek has joined #tripleo | 06:11 | |
| *** mrunge has joined #tripleo | 06:15 | |
| *** jcoufal has joined #tripleo | 06:20 | |
| SpamapS | lifeless: so, are we just plain down? | 06:25 |
| SpamapS | lifeless: also, trusty has the new mellanox driver already, in case you're feeling intrepid.. ;) | 06:25 |
| lifeless | SpamapS: we're just about up | 06:29 |
| lifeless | just configuring the geard broker now | 06:29 |
| SpamapS | lifeless: ok. Sorry I haven't been helpful. Busy Sunday. | 06:30 |
| lifeless | SpamapS: no wakkas | 06:30 |
| lifeless | life is | 06:31 |
| lifeless | greghaynes: 2014-03-31 06:36:52.569 25015 ERROR neutron.agent.linux.ovsdb_monitor [-] Error received from ovsdb monitor: 2014-03-31T06:36:52Z|00001|fatal_signal|WARN|terminating with signal 15 (Terminated) | 06:34 |
| lifeless | bah | 06:34 |
| lifeless | tchaypo: ^ | 06:34 |
| lifeless | ring any bells? | 06:34 |
| tchaypo | yerp. | 06:35 |
| tchaypo | restart neutron-openvswitch-agent, after correcting for spelling | 06:35 |
| lifeless | I get that when I restat noa | 06:35 |
| tchaypo | restarting nova triggers that error? | 06:36 |
| tchaypo | or restarting nova gives you the neutron agent restart for free | 06:36 |
| lifeless | neutron openvswitch agent | 06:36 |
| tchaypo | ? | 06:36 |
| tchaypo | ah, the latter. | 06:36 |
| lifeless | nothing to do with nova | 06:37 |
| lifeless | just when I restart n-o-a I get that logged | 06:37 |
| tchaypo | I squeaked a bit in openstack-neutron today and mestery has grabbed https://bugs.launchpad.net/tripleo/+bug/1290486 | 06:37 |
| uvirtbot | Launchpad bug 1290486 in tripleo "neutron-openvswitch-agent must be restarted after ovsdb-server failure in order to pass traffic" [Critical,In progress] | 06:37 |
| tchaypo | ah, interesting. | 06:38 |
| tchaypo | I think that means that your ovsdb-server process terminated with sig 15 | 06:39 |
| tchaypo | you might need to restart that (service openvswitch-switch restart) and then restart n_o_a | 06:39 |
| lifeless | byebye network connection | 06:40 |
| * tchaypo waves | 06:40 | |
| lifeless | service openvswitch-switch restart | 06:40 |
| lifeless | 2014-03-31 06:42:35.080 25453 ERROR neutron.agent.linux.ovsdb_monitor [-] Error received from ovsdb monitor: ovsdb-client: unix:/var/run/openvswitch/db.sock: receive failed (End of file) | 06:40 |
| lifeless | rvice neutron-openvswitch-agent restart | 06:40 |
| lifeless | 2014-03-31 06:42:59.489 25453 CRITICAL neutron [-] Trying to re-send() an already-triggered event. | 06:40 |
| lifeless | neutron-openvswitch-agent stop/waiting | 06:40 |
| *** jcoufal has quit IRC | 06:40 | |
| *** rpodolyaka1 has joined #tripleo | 06:41 | |
| rpodolyaka1 | morning all | 06:41 |
| lifeless | morning | 06:43 |
| lifeless | SpamapS: currently debugging overlay network fail | 06:43 |
| tchaypo | did it finish shutting down/restarting? | 06:46 |
| lifeless | tchaypo: apparently | 06:46 |
| lifeless | pings from network node to vm aren't reaching the vm | 06:46 |
| lifeless | looking for tagging rule failures now | 06:46 |
| SpamapS | lifeless: my experience with that when rebooting compute nodes for something was to restart neutron-openvswitch-agent after a few minutes. (source: very cloudy memory) | 06:47 |
| lifeless | SpamapS: yes | 06:47 |
| tchaypo | that BFI approach has been tried :) | 06:47 |
| lifeless | interestingly arps are getting through | 06:48 |
| lifeless | how bizarre | 06:48 |
| tchaypo | *cue soundtrack* | 06:48 |
| lifeless | 06:50:26.058055 fa:16:3e:c6:90:58 > fa:16:3e:ea:7d:01, ethertype ARP (0x0806), length 42: Request who-has 10.0.0.4 tell 10.0.0.1, length 28 | 06:48 |
| lifeless | 06:50:26.059257 fa:16:3e:ea:7d:01 > fa:16:3e:c6:90:58, ethertype ARP (0x0806), length 42: Reply 10.0.0.4 is-at fa:16:3e:ea:7d:01, length 28 | 06:48 |
| tchaypo | although I'm sure you meant to say "how bizarp" | 06:48 |
| SpamapS | oooo baby | 06:53 |
| SpamapS | its making me crazy | 06:53 |
| SpamapS | every time I look around.. IP's in my space. | 06:53 |
| lifeless | bah | 06:58 |
| lifeless | security groups | 06:58 |
| lifeless | ok broke is up | 07:00 |
| lifeless | deploying testenvs now | 07:04 |
| tchaypo | Did I just spam the channel with the news that I'm back? | 07:07 |
| *** matsuhashi has quit IRC | 07:08 | |
| tchaypo | if so I'll take weechat out the back and have a quiet word | 07:08 |
| *** matsuhashi has joined #tripleo | 07:08 | |
| *** nosnos_ has quit IRC | 07:09 | |
| *** nosnos has joined #tripleo | 07:10 | |
| lifeless | tchaypo: no | 07:10 |
| tchaypo | phew | 07:11 |
| *** bauzas has joined #tripleo | 07:14 | |
| lifeless | incoming | 07:16 |
| openstackgerrit | lifeless proposed a change to openstack/tripleo-image-elements: Performance tweaks for testenv deploy script. https://review.openstack.org/84073 | 07:16 |
| openstackgerrit | lifeless proposed a change to openstack/tripleo-image-elements: Remove options not present in the heat template. https://review.openstack.org/84074 | 07:16 |
| openstackgerrit | lifeless proposed a change to openstack/tripleo-image-elements: Fixup HP region testenv config. https://review.openstack.org/84075 | 07:16 |
| openstackgerrit | lifeless proposed a change to openstack/tripleo-image-elements: Tune deploy-ci-overcloud a little. https://review.openstack.org/84076 | 07:16 |
| *** ifarkas has joined #tripleo | 07:18 | |
| *** rpodolyaka1 has quit IRC | 07:20 | |
| *** rpodolyaka1 has joined #tripleo | 07:21 | |
| *** matsuhashi has quit IRC | 07:22 | |
| *** hashar has joined #tripleo | 07:24 | |
| xuhaiwei | when i was creating the mirror, I got this error, http://paste.openstack.org/show/74636/ , what is the problem? | 07:24 |
| tchaypo | xuhaiwei: "Connection to pypi.python.org timed out." | 07:25 |
| lifeless | its means you had the same problem you had when building images | 07:25 |
| lifeless | but the thing about the mirror | 07:25 |
| lifeless | is you can keep re-running that command | 07:25 |
| lifeless | until it completes | 07:25 |
| lifeless | then you'll have a local mirror | 07:25 |
| xuhaiwei | ... | 07:26 |
| xuhaiwei | I have kept running it several times | 07:26 |
| xuhaiwei | I can open it from the browser | 07:27 |
| openstackgerrit | lifeless proposed a change to openstack/tripleo-image-elements: Fixup HP region testenv config. https://review.openstack.org/84075 | 07:27 |
| openstackgerrit | lifeless proposed a change to openstack/tripleo-image-elements: Tune deploy-ci-overcloud a little. https://review.openstack.org/84076 | 07:27 |
| lifeless | xuhaiwei: yes, but your browser is making one request. building an image (or a mirror) requires hundreds of requests. | 07:28 |
| tchaypo | opening it from the browser is one connection; when you do a build, you need to open lots of connections to download lots of things | 07:28 |
| *** matsuhashi has joined #tripleo | 07:28 | |
| xuhaiwei | oh, OK, i will do it | 07:29 |
| lifeless | hmm | 07:37 |
| lifeless | sda 0.00 408.60 0.00 320.60 0.00 58037.60 362.06 142.03 401.64 0.00 401.64 3.12 100.00 | 07:37 |
| *** ramishra_ has quit IRC | 07:38 | |
| lifeless | deploys and diskIO :P | 07:38 |
| lifeless | I may have said before the lack of caching is an issue | 07:38 |
| *** jistr has joined #tripleo | 07:40 | |
| lifeless | jistr: morning ;) | 07:41 |
| jistr | lifeless: hi :) | 07:41 |
| lifeless | and now the deploy is running - its bottlenecked at sdb 0.00 25.40 0.00 97.80 0.00 49984.00 1022.17 1.58 16.65 0.00 16.65 9.96 97.36 | 07:42 |
| lifeless | so I suspect we're not saturating the network; I wonder if some tgt option changed | 07:42 |
| lifeless | (we shouldbe able to run way ahead into cache on the target machine) | 07:42 |
| *** jprovazn has joined #tripleo | 07:44 | |
| *** martyntaylor has joined #tripleo | 07:46 | |
| jistr | lifeless: re CLI for os-cloud-config -- you meant CLI for what? currently there's just a bit of keystone init in the master - creation of users, projects, domains etc. | 07:46 |
| *** matsuhashi has quit IRC | 07:46 | |
| *** matsuhashi has joined #tripleo | 07:47 | |
| lifeless | jistr: exactly | 07:47 |
| lifeless | jistr: that replaces init-keystone (core stuff only) and then the swift and heat users we add | 07:47 |
| openstackgerrit | Ghe Rivero proposed a change to openstack/tripleo-incubator: Allow to set seed node cpus and memory https://review.openstack.org/84078 | 07:47 |
| *** akrivoka has joined #tripleo | 07:48 | |
| lifeless | jistr: thats duplicate with incubator - and the goal of os-cloud-config is to be the common code for seed bringup, (CLI) and undercloud+overcloud bringup (tuskar) | 07:48 |
| jistr | lifeless: ok. yeah anyone can do the CLI :) Unfortunately i'm not able to work on patches for TripleO at the moment, at least this week. Re use within incubator -- i wondered whether we should run python -c '...' from incubator scripts, but that's probably not as nice as having proper CLI interface. | 07:50 |
| lifeless | jistr: I believe StevenK is going to be doing the register-nodes stuff | 07:50 |
| GheRivero | morning all | 07:50 |
| lifeless | jistr: I'll see if I can convince him to do a init-keystone, init-heat, init-swift CLI entrypoints driveby | 07:51 |
| lifeless | GheRivero: morning | 07:51 |
| jistr | lifeless: excellent, thanks :) | 07:51 |
| StevenK | lifeless: Moi? | 07:52 |
| StevenK | lifeless: Considered me convinced, I shall do so tomorrow. | 07:53 |
| jistr | StevenK: thanks :) | 07:53 |
| lifeless | {"error": {"message": "The request you have made requires authentication.", "code": 401, "title": "Unauthorized"}} | 07:56 |
| lifeless | STAB | 07:56 |
| lifeless | today has been an exercise in debugg | 07:56 |
| lifeless | ing | 07:56 |
| StevenK | lifeless: And stabbing | 07:56 |
| *** jcoufal has joined #tripleo | 07:58 | |
| *** mkerrin has joined #tripleo | 07:59 | |
| *** akuznetsov has quit IRC | 07:59 | |
| *** akuznetsov has joined #tripleo | 07:59 | |
| lifeless | ah | 08:00 |
| lifeless | forgot to add tripleo-ci to the nodepool tenant | 08:01 |
| *** shardy has joined #tripleo | 08:02 | |
| lifeless | ok | 08:05 |
| lifeless | that should now get us up (once we get infra patch deployed | 08:05 |
| *** akuznetsov has quit IRC | 08:06 | |
| *** derekh has joined #tripleo | 08:09 | |
| *** akrivoka has quit IRC | 08:10 | |
| *** rpodolyaka1 has quit IRC | 08:10 | |
| *** ramishra has joined #tripleo | 08:12 | |
| *** athomas has joined #tripleo | 08:12 | |
| *** jcoufal has quit IRC | 08:12 | |
| *** jcoufal has joined #tripleo | 08:13 | |
| *** rdopieralski has joined #tripleo | 08:13 | |
| *** yassine has joined #tripleo | 08:13 | |
| Ng | morning | 08:14 |
| GheRivero | morning Ng | 08:15 |
| *** rpodolyaka1 has joined #tripleo | 08:16 | |
| openstackgerrit | A change was merged to openstack/tripleo-incubator: Add clarification on required permissions https://review.openstack.org/83102 | 08:17 |
| *** e0ne has joined #tripleo | 08:17 | |
| *** rpodolyaka1 has quit IRC | 08:19 | |
| *** ramishra has quit IRC | 08:22 | |
| *** ramishra has joined #tripleo | 08:24 | |
| *** lucasagomes has joined #tripleo | 08:24 | |
| openstackgerrit | Nicholas Randon proposed a change to openstack/tripleo-incubator: Make it possible to pass real hardware in. https://review.openstack.org/83188 | 08:26 |
| openstackgerrit | Nicholas Randon proposed a change to openstack/tripleo-incubator: Brige interface to the seed. https://review.openstack.org/84083 | 08:27 |
| *** rpodolyaka1 has joined #tripleo | 08:28 | |
| lifeless | SpamapS: you might want to re-review https://review.openstack.org/#/c/83188/ w.r.t. my thoughts in patchset 2 | 08:28 |
| lifeless | GheRivero: Ng; hi! | 08:28 |
| lifeless | Ng: we had the ci-overcloud network node die again | 08:29 |
| lifeless | Ng: I haven't filed a ticket - I redeployed instead | 08:29 |
| lifeless | Ng: I've been sufficiently busy I only marked it as bad in nova | 08:29 |
| lifeless | Ng: :( | 08:29 |
| lifeless | Ng: monty005 | 08:29 |
| *** eghobo has quit IRC | 08:29 | |
| *** rpodolyaka1 has quit IRC | 08:32 | |
| *** akuznetsov has joined #tripleo | 08:32 | |
| Ng | lifeless: ok, I'll have a look and get a ticket filed | 08:32 |
| ProfFalken | hey all, is there an "openstack approved" webserver that we should be using as part of DIB elements (nginx/apache2/lighttpd/etc)? | 08:32 |
| lifeless | ProfFalken: we've used apache2 so far but in principle it should be pluggable | 08:34 |
| *** Daviey has joined #tripleo | 08:35 | |
| *** pblaho has joined #tripleo | 08:36 | |
| ProfFalken | lifeless: heh, yeah, although I've just found that if I'm running NGinx on my workstation then when I run devtest --trash-my-machine it craps out on an apache2 restart in the chroot because port 80 is in use on the host | 08:37 |
| *** giulivo has joined #tripleo | 08:37 | |
| lifeless | ProfFalken: huh, it shouldn't be restarting apache2 inside the chroot - no services are meant to be running there | 08:38 |
| ProfFalken | lifeless: it's because I'm testing something out and abusing the system ;) | 08:38 |
| lifeless | ah :) | 08:38 |
| ProfFalken | My element development cycle is basically "throw it all into install.d/foo then split it out into the correct places later" | 08:38 |
| ProfFalken | means I get a working element quite quickly without having to think too much! | 08:39 |
| ProfFalken | anyway, from an Icinga/Check_MK point of view, it looks like Apache2 has been chosen as "the one true way" so I'll keep cracking on with that | 08:39 |
| lifeless | I admire your bravery :) | 08:39 |
| ProfFalken | :D | 08:39 |
| ProfFalken | not bravery, cowardice - not ready to tackle HOT and DIB variables yet! :) | 08:40 |
| lifeless | derekh: morning | 08:43 |
| derekh | lifeless: good morning | 08:43 |
| lifeless | derekh: HP region ci-overcloud died in the weekend | 08:44 |
| lifeless | derekh: disk read errors and on reboot hangs on CPU 1 during boot | 08:44 |
| lifeless | derekh: I've redeployed the entire cloud | 08:44 |
| lifeless | derekh: and te broker | 08:44 |
| lifeless | derekh: just trying to get testenvs up now | 08:44 |
| lifeless | derekh: we need to get infra to land a patch before we'll be live again | 08:44 |
| lifeless | derekh: https://review.openstack.org/#/c/84061/ | 08:45 |
| lifeless | derekh: and hell, just look at https://review.openstack.org/#/dashboard/4190 :) | 08:45 |
| derekh | lifeless: ok, will do | 08:46 |
| lifeless | for now I'm poking at testenv, will let yo know when I stop | 08:46 |
| derekh | lifeless: ok, sounds good, will take over when your done | 08:47 |
| *** akrivoka has joined #tripleo | 08:48 | |
| openstackgerrit | Ghe Rivero proposed a change to openstack/tripleo-incubator: Allow to set seed node cpus and memory https://review.openstack.org/84078 | 08:49 |
| *** giulivo has quit IRC | 08:49 | |
| *** giulivo has joined #tripleo | 08:49 | |
| *** giulivo has quit IRC | 08:55 | |
| *** markmc has joined #tripleo | 08:56 | |
| *** giulivo has joined #tripleo | 08:58 | |
| *** jp_at_hp has joined #tripleo | 08:59 | |
| *** jp_at_hp has quit IRC | 08:59 | |
| *** jp_at_hp has joined #tripleo | 08:59 | |
| *** untriaged-bot has joined #tripleo | 09:00 | |
| untriaged-bot | Untriaged bugs so far: | 09:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1290488 | 09:00 |
| uvirtbot | Launchpad bug 1290488 in tripleo "Baremetal: Invalid credentials" [Undecided,Incomplete] | 09:00 |
| *** untriaged-bot has quit IRC | 09:00 | |
| *** jcoufal has quit IRC | 09:01 | |
| *** xuhaiwei has quit IRC | 09:03 | |
| derekh | lifeless: I can't ssh to the hp undercloud for some reason | 09:07 |
| lifeless | derekh: I'm currently sshed in | 09:07 |
| lifeless | derekh: what IP are you coming from ? | 09:07 |
| derekh | lifeless: 93.107.239.19 | 09:08 |
| lifeless | I see traffix | 09:08 |
| *** e0ne_ has joined #tripleo | 09:08 | |
| lifeless | 09:08:32.887855 IP 93.107.239.19.35778 > 138.35.77.3.22: Flags [S], seq 2346726437, win 29200, options [mss 1420,sackOK,TS val 3797520 ecr 0,nop,wscale 7], length 0 | 09:08 |
| lifeless | 09:08:32.887878 IP 138.35.77.3.22 > 93.107.239.19.35778: Flags [S.], seq 1694810703, ack 2346726438, win 14480, options [mss 1460,sackOK,TS val 2643115703 ecr 3794512,nop,wscale 7], length 0 | 09:09 |
| lifeless | 09:08:33.278791 IP 138.35.77.3.22 > 93.107.239.19.35778: Flags [S.], seq 1694810703, ack 2346726438, win 14480, options [mss 1460,sackOK,TS val 2643115801 ecr 3794512,nop,wscale 7], length 0 | 09:09 |
| lifeless | 09:08:36.895989 IP 93.107.239.19.35778 > 138.35.77.3.22: Flags [S], seq 2346726437, win 29200, options [mss 1420,sackOK,TS val 3801528 ecr 0,nop,wscale 7], length 0 | 09:09 |
| openstackgerrit | therese-mchale proposed a change to openstack/tripleo-image-elements: Make MySQL root password configurable https://review.openstack.org/81392 | 09:09 |
| lifeless | derekh: I see pings and replies | 09:10 |
| derekh | lifeless: the ACK,s arn't getting back to me | 09:10 |
| derekh | 10:10:08.449698 IP 192.168.1.112.35779 > 138.35.77.3.22: Flags [S], seq 1834679978, win 29200, options [mss 1460,sackOK,TS val 3893175 ecr 0,nop,wscale 7], length 0 | 09:10 |
| derekh | 10:10:09.449889 IP 192.168.1.112.35779 > 138.35.77.3.22: Flags [S], seq 1834679978, win 29200, options [mss 1460,sackOK,TS val 3894176 ecr 0,nop,wscale 7], length 0 | 09:10 |
| derekh | 10:10:11.453884 IP 192.168.1.112.35779 > 138.35.77.3.22: Flags [S], seq 1834679978, win 29200, options [mss 1460,sackOK,TS val 3896180 ecr 0,nop,wscale 7], length 0 | 09:10 |
| lifeless | derekh: no ack! | 09:10 |
| lifeless | derekh: did the pongs? | 09:11 |
| derekh | lifeless: nope, no pongs | 09:11 |
| lifeless | derekh: does ci-overcloud let you in ? | 09:11 |
| *** e0ne has quit IRC | 09:12 | |
| lifeless | 09:11:51.207520 IP 138.35.77.3 > 93.107.239.19: ICMP echo reply, id 6996, seq 27, length 64 | 09:12 |
| lifeless | 09:11:52.206937 IP 93.107.239.19 > 138.35.77.3: ICMP echo request, id 6996, seq 28, length 64 | 09:12 |
| lifeless | 09:11:52.206961 IP 138.35.77.3 > 93.107.239.19: ICMP echo reply, id 6996, seq 28, length 64 | 09:12 |
| *** gcha has joined #tripleo | 09:12 | |
| derekh | lifeless: Yup, I can ssh to ci-overcloud.tripleo.org | 09:13 |
| lifeless | how bizaare | 09:13 |
| lifeless | local firewall or something? | 09:17 |
| SpamapS | new SSH key? | 09:19 |
| SpamapS | lifeless: so old disk was kaput? | 09:20 |
| * SpamapS should really be in bed | 09:20 | |
| lifeless | SpamapS: no, I think its mb | 09:26 |
| lifeless | SpamapS: see under 'locks up on boot' | 09:26 |
| lifeless | right now nova-compute is off in lala land not deploying the testenv | 09:27 |
| lifeless | derekh: so I have a good testenv image | 09:27 |
| openstackgerrit | Ana Krivokapic proposed a change to openstack/tuskar-ui: Fix exception logging https://review.openstack.org/83448 | 09:27 |
| lifeless | derekh: it was deploying before but I hadn't setup keystone in ci-overcloud right (have added to test-cluster notes) | 09:27 |
| lifeless | derekh: I'm deploying with ./bin/deploy-testenv configs/te_hp1rc | 09:28 |
| lifeless | derekh: from the root screen session | 09:28 |
| derekh | lifeless: ok | 09:31 |
| lifeless | but the 10 testenv boxes have been in state spawning for 20m now... | 09:31 |
| lifeless | not right ;) | 09:31 |
| *** rpodolyaka1 has joined #tripleo | 09:32 | |
| derekh | o dear | 09:33 |
| *** rpodolyaka1 has quit IRC | 09:35 | |
| lifeless | derekh: yah. Can I hand over? | 09:36 |
| derekh | lifeless: yup, will see what I can do | 09:37 |
| derekh | lifeless: can I get to undercloud node from the ci-overcloud, still can't ssh directly to undercloud ? | 09:38 |
| openstackgerrit | Alexis Lee proposed a change to openstack/diskimage-builder: Add ability to add extra apt keys https://review.openstack.org/83596 | 09:45 |
| lifeless | derekh: ssh -A heat-admin@ci-overcloud.tripleo.org ssh cd-undercloud.tripleo.org ? | 09:48 |
| derekh | lifeless: nope, not getting a ssh prompt from the undercloud at all | 09:51 |
| *** jcoufal has joined #tripleo | 09:51 | |
| lifeless | let me make sure I do :) | 09:53 |
| lifeless | ok reproduced | 09:57 |
| lifeless | clearly something naffed | 09:57 |
| lifeless | ok, I'm going to reboot it | 09:58 |
| derekh | lifeless: afraid I don't think I can help much until I can get in | 09:58 |
| derekh | lifeless: ok | 09:58 |
| lifeless | fingers crossed | 09:58 |
| lifeless | I may have just made it worse :P | 09:58 |
| derekh | we'll soon find out | 09:59 |
| openstackgerrit | Nicholas Randon proposed a change to openstack/tripleo-incubator: Brige interface to the seed. https://review.openstack.org/84083 | 10:02 |
| openstackgerrit | A change was merged to openstack/tuskar-ui: Fix exception logging https://review.openstack.org/83448 | 10:02 |
| lifeless | oh, I hope the seed VM is running. well, yet another thing to find out | 10:03 |
| lifeless | apparently not... | 10:04 |
| lifeless | recursing | 10:04 |
| *** lxsli has quit IRC | 10:07 | |
| lifeless | oh wow | 10:07 |
| lifeless | manual dnsmask | 10:07 |
| lifeless | enabled | 10:07 |
| lifeless | fingers (on the seed) - predates our use of neutron | 10:08 |
| *** lxsli has joined #tripleo | 10:08 | |
| *** michchap has quit IRC | 10:11 | |
| lifeless | ok tftp booted, into kernel | 10:11 |
| lifeless | stopping wildcard dnsmasq | 10:12 |
| openstackgerrit | Nicholas Randon proposed a change to openstack/tripleo-incubator: Brige interface to the seed. https://review.openstack.org/84083 | 10:15 |
| openstackgerrit | Nicholas Randon proposed a change to openstack/tripleo-incubator: Make it possible to pass real hardware in. https://review.openstack.org/83188 | 10:17 |
| lifeless | had to run ensure-bridge once by hand | 10:18 |
| lifeless | now running o-c-c --force --one | 10:18 |
| *** gcha has quit IRC | 10:20 | |
| *** yamahata has quit IRC | 10:21 | |
| lifeless | heh WAT | 10:22 |
| lifeless | 169.154.169.154 via 10.10.16.168 dev br-ctlplane | 10:22 |
| lifeless | now seed metadata service is in go-slow mode | 10:26 |
| lifeless | /dev/vda1 30G 28G 877M 97% / | 10:27 |
| lifeless | I smell massive keystone db | 10:27 |
| lifeless | | 762884 | | 10:27 |
| *** rpodolyaka1 has joined #tripleo | 10:33 | |
| *** jcoufal has quit IRC | 10:33 | |
| lifeless | heat stack-list hanging now | 10:34 |
| lifeless | on the seed | 10:34 |
| *** rpodolyaka1 has quit IRC | 10:37 | |
| openstackgerrit | Ryan Moore proposed a change to openstack/tripleo-image-elements: Restructure the nova.conf to match documentation https://review.openstack.org/83821 | 10:38 |
| *** jcoufal has joined #tripleo | 10:43 | |
| *** cd-undercloud has joined #tripleo | 10:47 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 10:47 |
| *** cd-undercloud has quit IRC | 10:47 | |
| *** cd-undercloud has joined #tripleo | 10:47 | |
| cd-undercloud | ************** overcloud complete status=1 ************ | 10:47 |
| *** cd-undercloud has quit IRC | 10:47 | |
| derekh | lifeless: Now that I can ssh to undercloud, want me to do anything or are you still working on it? | 10:53 |
| lifeless | derekh: I'm poking at the seed | 10:54 |
| lifeless | derekh: which we haven't set you up with access to ... but lets do that | 10:54 |
| derekh | k | 10:54 |
| lifeless | actually | 10:54 |
| lifeless | you should have | 10:54 |
| *** e0ne_ has quit IRC | 10:54 | |
| lifeless | pmed you the address | 10:54 |
| lifeless | root@ that | 10:54 |
| derekh | lifeless: ok, I'm in | 10:55 |
| lifeless | ok | 10:55 |
| lifeless | so the current problem is | 10:55 |
| lifeless | wget -O- http://169.254.169.254/latest | 10:55 |
| lifeless | on the undercloud hangs | 10:55 |
| lifeless | the seed is seeing the requeset | 10:55 |
| lifeless | and going off into lala land | 10:55 |
| lifeless | I was just about to try disabling conductor and restarting the api | 10:55 |
| lifeless | a separate symptom | 10:55 |
| lifeless | is that heat stack-list run on the seed itself hangs | 10:56 |
| lifeless | hmm, maybe try restarting rabbit first | 10:56 |
| lifeless | nup | 10:56 |
| lifeless | wget -O- http://169.254.169.254/latest | 10:56 |
| lifeless | --2014-03-31 10:56:25-- http://169.254.169.254/latest | 10:56 |
| lifeless | Connecting to 169.254.169.254:80... connected. | 10:56 |
| lifeless | HTTP request sent, awaiting response... | 10:56 |
| lifeless | so I was about to disable conductor as I mentioned, and then start running the api itneractively with pdb breakpoints :/ | 10:57 |
| derekh | lifeless: ok | 10:57 |
| lifeless | once wget -O- http://169.254.169.254/latest from the undercloud works | 10:57 |
| lifeless | then os-collect-config --force --one | 10:57 |
| lifeless | should get it back working | 10:57 |
| lifeless | and then we can deploy the testenv | 10:58 |
| lifeless | and profit | 10:58 |
| *** vkozhukalov has joined #tripleo | 10:58 | |
| *** andreaf has joined #tripleo | 10:58 | |
| lifeless | ahha | 10:58 |
| lifeless | disabling conductor did it | 10:58 |
| derekh | lifeless: ok, weird, why would that work | 10:59 |
| lifeless | well there's presumably something crooked in the seed | 11:00 |
| lifeless | os-c-c is now hanging on heat I *think* | 11:00 |
| lifeless | yeah | 11:00 |
| lifeless | we need to fix heat | 11:00 |
| lifeless | IIRC heat stack-list is served from heat-engine by heat-api | 11:01 |
| lifeless | so I'd be looking at something rabbit related | 11:02 |
| lifeless | derekh: GL, I'll check in in ~7hrs | 11:02 |
| derekh | lifeless: ok, fingers crossed :-) | 11:03 |
| derekh | lifeless: will see how far I can get | 11:03 |
| lifeless | once heat stack-list works I expect os-collect-config will be able to read the heat metadata | 11:04 |
| derekh | lifeless: ok, makes sense | 11:04 |
| lifeless | os-collect-config --force --one -v | 11:05 |
| lifeless | 2014-03-31 11:00:34.466 12017 INFO requests.packages.urllib3.connectionpool [-] Starting new HTTP connection (1): 169.254.169.254 | 11:05 |
| lifeless | 2014-03-31 11:00:35.541 12017 INFO requests.packages.urllib3.connectionpool [-] Starting new HTTP connection (1): 10.10.16.168 | 11:05 |
| lifeless | (thats that the undercloud shows atm) | 11:05 |
| openstackgerrit | Nicholas Randon proposed a change to openstack/tripleo-incubator: Brige interface to the seed. https://review.openstack.org/84083 | 11:08 |
| lifeless | derekh: right past midnight - I'm thoroughly out :) | 11:10 |
| derekh | lifeless: ok, have a good night, I'll keep at it | 11:11 |
| *** michchap has joined #tripleo | 11:11 | |
| *** rlandy has joined #tripleo | 11:12 | |
| *** slagle has joined #tripleo | 11:12 | |
| *** slagle has quit IRC | 11:25 | |
| *** matsuhashi has quit IRC | 11:26 | |
| *** nosnos has quit IRC | 11:27 | |
| *** slagle has joined #tripleo | 11:32 | |
| *** e0ne has joined #tripleo | 11:34 | |
| openstackgerrit | A change was merged to openstack/tripleo-incubator: Document remote-host in test environment json https://review.openstack.org/82946 | 11:34 |
| openstackgerrit | Jon-Paul Sullivan proposed a change to openstack/tripleo-image-elements: Standardise manifest creation and retrieval https://review.openstack.org/83850 | 11:40 |
| openstackgerrit | Jon-Paul Sullivan proposed a change to openstack/diskimage-builder: Standardise manifest creation and retrieval https://review.openstack.org/83848 | 11:40 |
| *** lucasagomes is now known as lucas-hungry | 11:40 | |
| openstackgerrit | A change was merged to openstack/os-collect-config: Add diagram to os-collect-config README.rst https://review.openstack.org/83115 | 11:41 |
| *** jprovazn has quit IRC | 11:42 | |
| *** e0ne has quit IRC | 11:44 | |
| *** e0ne has joined #tripleo | 11:44 | |
| *** mrunge has quit IRC | 11:46 | |
| *** spzala has joined #tripleo | 11:57 | |
| derekh | trying to debug cd-seed, anybody with cd-admin access want to take a look gotta pop away for a little bit | 12:02 |
| derekh | basically "heat list" hangs and trying to find out why, the cd-undercloud is blocked on getting this working | 12:03 |
| derekh | looks like the problem with RPC between heat-api and heat-engine | 12:04 |
| *** spzala has quit IRC | 12:04 | |
| *** ramishra_ has joined #tripleo | 12:10 | |
| *** ramishra has quit IRC | 12:13 | |
| *** jpeeler has joined #tripleo | 12:19 | |
| openstackgerrit | Nicholas Randon proposed a change to openstack/tripleo-incubator: SSH key and virtual_power_driver not used on H/W https://review.openstack.org/83770 | 12:22 |
| openstackgerrit | Nicholas Randon proposed a change to openstack/tripleo-incubator: Make it possible to pass real hardware in. https://review.openstack.org/83188 | 12:22 |
| openstackgerrit | Nicholas Randon proposed a change to openstack/tripleo-incubator: SSH key and virtual_power_driver not used on H/W https://review.openstack.org/83770 | 12:24 |
| openstackgerrit | Nicholas Randon proposed a change to openstack/tripleo-incubator: SSH key and virtual_power_driver not used on H/W https://review.openstack.org/83770 | 12:25 |
| *** rpodolyaka1 has joined #tripleo | 12:32 | |
| *** rbrady has joined #tripleo | 12:33 | |
| openstackgerrit | Nicholas Randon proposed a change to openstack/tripleo-incubator: Brige interface to the seed. https://review.openstack.org/84083 | 12:34 |
| *** rpodolyaka1 has quit IRC | 12:37 | |
| *** lucas-hungry is now known as lucasagomes | 12:41 | |
| *** jdob has joined #tripleo | 12:44 | |
| openstackgerrit | Nicholas Randon proposed a change to openstack/tripleo-incubator: Brige interface to the seed. https://review.openstack.org/84083 | 12:44 |
| *** gcha has joined #tripleo | 12:47 | |
| *** dprince has joined #tripleo | 12:49 | |
| *** yamahata has joined #tripleo | 12:52 | |
| *** akuznetsov has quit IRC | 12:52 | |
| dprince | derekh: so this is the fix for my ensure-bridge branch https://review.openstack.org/#/c/83867/. Once we have that fix we'll need to rebuild the test env's though... | 12:53 |
| derekh | dprince: ok, great, will take a look, trying to debug heat one the seed in the HP rack at the moment seems to broke | 12:55 |
| derekh | dprince: will take a look once I figure out whats up | 12:56 |
| dprince | derekh: no rush, more of just a heads up that we'll need to rebuild the testenv's (or manually apply this fix) once that change is in | 12:57 |
| *** sballe has joined #tripleo | 12:57 | |
| derekh | dprince: ok | 12:57 |
| *** hashar has quit IRC | 13:00 | |
| *** akuznetsov has joined #tripleo | 13:03 | |
| *** e0ne has quit IRC | 13:03 | |
| *** lblanchard has joined #tripleo | 13:03 | |
| *** e0ne has joined #tripleo | 13:04 | |
| *** vkozhukalov has quit IRC | 13:05 | |
| *** hashar has joined #tripleo | 13:05 | |
| *** jdob has quit IRC | 13:06 | |
| *** jdob has joined #tripleo | 13:06 | |
| *** vkozhukalov has joined #tripleo | 13:07 | |
| *** spzala has joined #tripleo | 13:10 | |
| derekh | hmm, running "rabbitmqctl list_connections" on the seed seems to produce a load of blocked connections, is this normal ? http://paste.openstack.org/show/74659/ | 13:10 |
| openstackgerrit | Nicholas Randon proposed a change to openstack/tripleo-incubator: Make it possible to pass real hardware in. https://review.openstack.org/83188 | 13:11 |
| derekh | ahhh | 13:14 |
| derekh | Disk free space insufficient. Free bytes:911364096 Limit:1000000000 | 13:14 |
| openstackgerrit | Nicholas Randon proposed a change to openstack/tripleo-incubator: Brige interface to the seed. https://review.openstack.org/84083 | 13:14 |
| dprince | derekh: we need some monitoring/metrics to tell us about these things | 13:16 |
| *** weshay has joined #tripleo | 13:16 | |
| derekh | dprince: yup, we do | 13:16 |
| *** morazi has joined #tripleo | 13:17 | |
| *** matty_du1s|gone is now known as matty_dubs | 13:25 | |
| *** julim has joined #tripleo | 13:27 | |
| *** ramishra_ has quit IRC | 13:28 | |
| *** ramishra_ has joined #tripleo | 13:30 | |
| *** ccrouch has joined #tripleo | 13:31 | |
| openstackgerrit | Andrea Rosa proposed a change to openstack/tripleo-image-elements: Allow settings for Nova Scheduler https://review.openstack.org/84131 | 13:31 |
| *** julim has quit IRC | 13:33 | |
| *** jdob_ has joined #tripleo | 13:35 | |
| *** funzo_ is now known as funzo | 13:36 | |
| *** julim has joined #tripleo | 13:36 | |
| openstackgerrit | Andrea Rosa proposed a change to openstack/tripleo-image-elements: Allow settings for Nova Scheduler https://review.openstack.org/84131 | 13:41 |
| *** jcoufal has quit IRC | 13:49 | |
| *** zigo has quit IRC | 13:50 | |
| *** akrivoka has quit IRC | 13:53 | |
| openstackgerrit | Alexis Lee proposed a change to openstack/diskimage-builder: Add ability to add extra apt keys https://review.openstack.org/83596 | 13:55 |
| *** noslzzp has joined #tripleo | 13:56 | |
| *** zigo has joined #tripleo | 13:58 | |
| *** lucasagomes has quit IRC | 13:59 | |
| *** lucasagomes has joined #tripleo | 13:59 | |
| *** ramishra has joined #tripleo | 14:01 | |
| *** ramishra_ has quit IRC | 14:02 | |
| *** lblanchard1 has joined #tripleo | 14:02 | |
| *** lblanchard has quit IRC | 14:04 | |
| *** rwsu has joined #tripleo | 14:06 | |
| *** ramishra_ has joined #tripleo | 14:08 | |
| openstackgerrit | Nicholas Randon proposed a change to openstack/tripleo-incubator: Bridge physical interface to the seed. https://review.openstack.org/84083 | 14:08 |
| *** ramishra has quit IRC | 14:09 | |
| *** bnemec has joined #tripleo | 14:09 | |
| *** slagle has quit IRC | 14:10 | |
| *** slagle has joined #tripleo | 14:12 | |
| *** yamahata has quit IRC | 14:19 | |
| *** yamahata has joined #tripleo | 14:19 | |
| openstackgerrit | A change was merged to openstack/tripleo-image-elements: Add mysql-mariadb-compat element https://review.openstack.org/78034 | 14:28 |
| openstackgerrit | A change was merged to openstack/tripleo-image-elements: Add no-op cinder-tgt element https://review.openstack.org/81867 | 14:28 |
| *** sballe has quit IRC | 14:29 | |
| jp_at_hp | dprince: regarding https://review.openstack.org/#/c/83188 - I think that one of the problems with TE_DATAFILE is that it combines information the user provides with generated information that ends up in the images. I think that this patch is starting to make amends to that problem. | 14:35 |
| *** ramishra_ has quit IRC | 14:36 | |
| dprince | jp_at_hp: okay. what do you mean by 'information that ends up in the images' | 14:36 |
| jp_at_hp | dprince: well I see the undercloud/overcloud password in there... | 14:37 |
| dprince | jp_at_hp: you mean in the testenv.json file? | 14:37 |
| jp_at_hp | y | 14:38 |
| *** jdob_ has quit IRC | 14:38 | |
| dprince | jp_at_hp: okay. So in that regard the file is being used (probably incorrectly) as a cache. | 14:38 |
| dprince | jp_at_hp: I think that is a bit of CI leakage (probably something that was stashed in there for CI) | 14:38 |
| dprince | jp_at_hp: if your goal is to split things up then, I'm slighly concerned about having to change things in multiple places then | 14:39 |
| jp_at_hp | dprince: hmmm, I would agree that the purpose of te_datafile seems to be a little muddy. perhaps lifeless can comment better on why he originated the nodes patch. I certainly can see a benefit in being able to supply a set of nodes independently from the other settings in that file. | 14:39 |
| * jp_at_hp gets the broken record out... | 14:40 | |
| dprince | jp_at_hp: I'm fine abstracting things, but this specific patch doesn't seem related to that goal IMO. And if we do get there I would see this as sort of the last step to finalize it | 14:40 |
| jp_at_hp | dprince: I would love to seethe https://etherpad.openstack.org/p/tripleo-devtest.sh-refactoring-blueprint and https://etherpad.openstack.org/p/tripleo-incubator-rationalise-ui designs fleshed out so that we all know where we should be heading... | 14:41 |
| *** jdob has quit IRC | 14:41 | |
| dprince | jp_at_hp: Would have been nice to discuss some of that at the meetup a few weeks back for sure... | 14:44 |
| *** hashar_ has joined #tripleo | 14:45 | |
| jp_at_hp | dprince: I think it was added after that, but yes - needs more eyes on it. I think lifeless posted them to the dev mailing list | 14:45 |
| NobodyCam | good morning TripleO | 14:45 |
| *** ramishra has joined #tripleo | 14:45 | |
| *** hashar has quit IRC | 14:48 | |
| *** mkerrin has quit IRC | 14:53 | |
| openstackgerrit | Loganathan Parthipan proposed a change to openstack/tripleo-incubator: Make libvirt volume pool configurable https://review.openstack.org/83409 | 14:54 |
| jang | dprince: yeah. It looked like testenv.json was becoming a grab-bag. The idea with that patch is to be able to do without --existing-environment and still have some clean inputs to the whole devtest process. | 14:54 |
| dprince | jang: I like that idea, but I don't thing we should dup all the settings (essentially causing a maintainence problem) | 14:56 |
| jang | There's a cleanup of testenv.json and a rationalisation of thoat whole interface going on - StevenK I think is doing a lot of work on it. lifeless' patch is a step in that direction. | 14:56 |
| jang | ah, there we probably view testenv,json differently. I don't see it as a duplication; I see testenv as a cache of data that is passed around by parts of the devtest process. | 14:56 |
| dprince | jang: perhaps we need testenv.json to be imutable, and then have another file for specific insteance cache stuff | 14:56 |
| jang | so the patch is for injecting some clean stuff into that at the start of the process. | 14:57 |
| jang | dprince: yes. And then we could rename testenv to nodes.json and have the cache called testenv :-) | 14:57 |
| jang | ... that's the moral effect of this, I think. | 14:57 |
| dprince | jang: I think we are saying the same thing then. :) | 14:57 |
| jang | we are. the question is, which file should be the junkpile. | 14:57 |
| jang | our impression, as newcomers learning the way the scripts work, was that that was what testenv was. | 14:58 |
| jang | because... let's face it... | 14:58 |
| dprince | jang: my confusion was around the fact that the docs sort of point you to create your own testenv.json (what I've been doing) | 14:58 |
| jang | yeah, but there are a lot of places where the UI process is "edit this file, source that file, run this script, source that script..." | 14:58 |
| *** jeblair_ is now known as jeblair | 14:59 | |
| jang | I think the general desire is to rationalise that and turn devtest into something that users can use a little more effectively. | 14:59 |
| *** untriaged-bot has joined #tripleo | 15:00 | |
| untriaged-bot | Untriaged bugs so far: | 15:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1290488 | 15:00 |
| *** untriaged-bot has quit IRC | 15:00 | |
| uvirtbot | Launchpad bug 1290488 in tripleo "Baremetal: Invalid credentials" [Undecided,Incomplete] | 15:00 |
| *** ramishra_ has joined #tripleo | 15:04 | |
| *** ramishra has quit IRC | 15:05 | |
| *** blamar has quit IRC | 15:07 | |
| openstackgerrit | Ladislav Smola proposed a change to openstack/tuskar-ui: Making overcloud init work https://review.openstack.org/83340 | 15:08 |
| *** CaptTofu has joined #tripleo | 15:09 | |
| *** mkerrin has joined #tripleo | 15:11 | |
| *** BadCub01 has joined #tripleo | 15:11 | |
| *** ramishra_ has quit IRC | 15:11 | |
| *** ramishra has joined #tripleo | 15:12 | |
| *** BadCub01 has left #tripleo | 15:12 | |
| *** BadCub01 has joined #tripleo | 15:13 | |
| *** bauzas has quit IRC | 15:13 | |
| *** newell has joined #tripleo | 15:14 | |
| *** ramishra has quit IRC | 15:16 | |
| derekh | SpamapS: do you know if the heat config on the cd-undercloud had any manual updates on it? heat isn't working and I have a feeling some config may have been overwritten by a o-r-c | 15:20 |
| openstackgerrit | A change was merged to openstack/tripleo-incubator: Add instructions for setting up sudo https://review.openstack.org/83238 | 15:21 |
| *** ramishra has joined #tripleo | 15:21 | |
| *** CaptTofu has quit IRC | 15:23 | |
| *** andreaf has quit IRC | 15:24 | |
| openstackgerrit | Nicholas Randon proposed a change to openstack/tripleo-incubator: Bridge physical interface to the seed. https://review.openstack.org/84083 | 15:28 |
| *** CaptTofu has joined #tripleo | 15:31 | |
| *** mkerrin has quit IRC | 15:31 | |
| *** ramishra has quit IRC | 15:36 | |
| *** geerdest has joined #tripleo | 15:37 | |
| *** UtahDave has joined #tripleo | 15:38 | |
| *** mkerrin has joined #tripleo | 15:38 | |
| openstackgerrit | A change was merged to openstack/python-tuskarclient: Changed role count assignment to use role name https://review.openstack.org/80052 | 15:38 |
| openstackgerrit | A change was merged to openstack/tripleo-image-elements: Install the check_mk Agent https://review.openstack.org/81485 | 15:39 |
| openstackgerrit | Dan Prince proposed a change to openstack/tripleo-incubator: Don't hard code the baremetal seed IP in seedrc https://review.openstack.org/83126 | 15:39 |
| openstackgerrit | Dan Prince proposed a change to openstack/tripleo-incubator: Write network configuration into seeds config.json https://review.openstack.org/83125 | 15:39 |
| openstackgerrit | Dan Prince proposed a change to openstack/tripleo-incubator: Make the baremetal-network configurable https://review.openstack.org/82327 | 15:39 |
| *** rdopieralski has quit IRC | 15:40 | |
| openstackgerrit | A change was merged to openstack/diskimage-builder: Fix "(None)" seed hostname on Debian https://review.openstack.org/77301 | 15:41 |
| *** ramishra has joined #tripleo | 15:49 | |
| *** gcha has quit IRC | 15:51 | |
| *** vkozhukalov has quit IRC | 15:53 | |
| *** CaptTofu has quit IRC | 15:56 | |
| *** hashar_ has quit IRC | 15:56 | |
| *** jcoufal has joined #tripleo | 15:58 | |
| SpamapS | derekh: the only manual update I did on heat.conf on cd-undercloud was to turn on debug sometimes. | 15:59 |
| *** eghobo has joined #tripleo | 15:59 | |
| SpamapS | derekh: what's broken? | 15:59 |
| * SpamapS has to go to an appointment in 2 minutes... but will look ASAP | 15:59 | |
| derekh | SpamapS: just found the problem | 16:00 |
| derekh | SpamapS: policy.json needed to be updated | 16:00 |
| SpamapS | derekh: ahh did we update heat to latest? | 16:01 |
| *** e0ne_ has joined #tripleo | 16:01 | |
| derekh | SpamapS: I am running on the working theory that it was updated at some stage but the o-a-c templates were left to the old version | 16:01 |
| derekh | SpamapS: so undid any configs that were put into /etc/heat | 16:02 |
| derekh | SpamapS: anyways heat appears to be working now, so I'll see that the next problem is :-) | 16:02 |
| SpamapS | derekh: yeah we've updated git but not the elements | 16:04 |
| *** ramishra has quit IRC | 16:04 | |
| *** e0ne has quit IRC | 16:04 | |
| SpamapS | derekh: oh the happy day when we have HA controllers and we can stop this circus ;) | 16:04 |
| derekh | SpamapS: it can't come too soon... | 16:04 |
| SpamapS | derekh: will hopefully get my template refactor in order this week which will get us quite a bit closer. | 16:06 |
| * SpamapS -> appointment | 16:07 | |
| *** cwolferh_ has joined #tripleo | 16:07 | |
| *** matty_dubs is now known as matty_dubs|lunch | 16:10 | |
| *** e0ne has joined #tripleo | 16:12 | |
| devananda | SpamapS: good morning! | 16:13 |
| devananda | d'oh. just missed him | 16:13 |
| devananda | ya'll shoud add your logs to logstash | 16:14 |
| *** sdake has joined #tripleo | 16:14 | |
| *** sdake has quit IRC | 16:14 | |
| *** sdake has joined #tripleo | 16:14 | |
| *** e0ne_ has quit IRC | 16:15 | |
| *** martyntaylor has quit IRC | 16:17 | |
| *** martyntaylor has joined #tripleo | 16:18 | |
| *** markmc has quit IRC | 16:18 | |
| *** CaptTofu has joined #tripleo | 16:24 | |
| devananda | SpamapS, lifeless: as a fix for the exception-driver-not-found issue, would it be sufficient (temporarily) for ya'll to install python-seamicroclient alongside ironic? that'll give us time to implement a better fix once Juno opens | 16:34 |
| *** martyntaylor has quit IRC | 16:36 | |
| openstackgerrit | Petr Blaho proposed a change to openstack/tuskar: Updates gettextutils module from oslo-incubator https://review.openstack.org/84171 | 16:37 |
| openstackgerrit | Petr Blaho proposed a change to openstack/tuskar: Updates test module from oslo-incubator https://review.openstack.org/84172 | 16:37 |
| openstackgerrit | Petr Blaho proposed a change to openstack/tuskar: Updates local module from oslo-incubator https://review.openstack.org/84173 | 16:37 |
| openstackgerrit | Petr Blaho proposed a change to openstack/tuskar: Updates timeutils module from oslo-incubator https://review.openstack.org/84174 | 16:37 |
| openstackgerrit | Petr Blaho proposed a change to openstack/tuskar: Updates importutils module from oslo-incubator https://review.openstack.org/84175 | 16:37 |
| openstackgerrit | Petr Blaho proposed a change to openstack/tuskar: Updates jsonutils module from oslo-incubator https://review.openstack.org/84176 | 16:37 |
| openstackgerrit | Petr Blaho proposed a change to openstack/tuskar: Updates excutils module from oslo-incubator https://review.openstack.org/84177 | 16:37 |
| openstackgerrit | Petr Blaho proposed a change to openstack/tuskar: Updates fileutils module from oslo-incubator https://review.openstack.org/84178 | 16:37 |
| openstackgerrit | Petr Blaho proposed a change to openstack/tuskar: Updates log module from oslo-incubator https://review.openstack.org/84179 | 16:37 |
| *** e0ne has quit IRC | 16:39 | |
| openstackgerrit | Dan Prince proposed a change to openstack/tripleo-incubator: overcloud: Look for notCompute or controller https://review.openstack.org/84180 | 16:40 |
| *** jb11211 has joined #tripleo | 16:42 | |
| derekh | lifeless: gotta run in a few minutes, quick update | 16:48 |
| dprince | derekh: I'm happy for you to do the notCompute stuff... Just keep reviewing code and see people adding more NOTCOMPUTE's so I figured it was time. | 16:49 |
| derekh | lifeless: heat problem on the seed was with rabbit, it didn't like having less then 1G of free disk space | 16:49 |
| dprince | derekh: Have you done the templates yet? (I've got those too) | 16:49 |
| *** jdob has joined #tripleo | 16:50 | |
| derekh | dprince: either is fine by me, I had done the heat template stuff but havn't yet tested each commit yet to ensure it would all get trough CI properly. | 16:52 |
| dprince | derekh: well, we need the incubator change landed first, then the templates | 16:52 |
| dprince | derekh: then there is a bunch of doc cleanup | 16:53 |
| openstackgerrit | Petr Blaho proposed a change to openstack/tuskar: Updates lockutils module from oslo-incubator https://review.openstack.org/84184 | 16:53 |
| openstackgerrit | Petr Blaho proposed a change to openstack/tuskar: Updates fixture module from oslo-incubator https://review.openstack.org/84185 | 16:53 |
| derekh | dprince: yup, we do, I was just holding off until I ensured the incubator change would work with the heat change, anyways I can send you what I have or send it up later either is good with me | 16:54 |
| derekh | lifeless: then it had 2 seperate problems with heat on the undercloud | 16:54 |
| derekh | lifeless: it looks liek the heat config on undercloud had manual updates that were not reflected in the o-a-c templates (and then got zapped by o-a-c) | 16:55 |
| derekh | lifeless: I had to change paste.filter_factory = heat.common.auth_token:filter_factory -> keystoneclient.middleware.auth_token:filter_factory | 16:55 |
| derekh | lifeless: and updated policy.json | 16:56 |
| derekh | lifeless: with that undercloud heat list started working but now can't delete the testenv stack | 16:56 |
| *** martyntaylor has joined #tripleo | 16:56 | |
| *** matty_dubs|lunch is now known as matty_dubs | 16:57 | |
| derekh | lifeless: so current status is, I can't delete the testenv stack getting this in the heat-engine.log http://paste.openstack.org/show/74672/ | 16:57 |
| *** pblaho has quit IRC | 16:58 | |
| *** CaptTofu has quit IRC | 16:58 | |
| openstackgerrit | Dan Prince proposed a change to openstack/tripleo-incubator: overcloud: Look for notCompute or controller https://review.openstack.org/84180 | 17:00 |
| *** TravT has joined #tripleo | 17:03 | |
| *** derekh has quit IRC | 17:04 | |
| *** martyntaylor has left #tripleo | 17:05 | |
| * SpamapS back | 17:06 | |
| *** TravT has quit IRC | 17:07 | |
| openstackgerrit | Kiall Mac Innes proposed a change to openstack/tripleo-image-elements: mysql-common's 50-mysql-users script requires python MySQLdb https://review.openstack.org/83969 | 17:07 |
| *** TravT has joined #tripleo | 17:07 | |
| openstackgerrit | Kiall Mac Innes proposed a change to openstack/tripleo-image-elements: mysql-common should not require a userhandle https://review.openstack.org/83970 | 17:08 |
| *** rwsu has quit IRC | 17:11 | |
| *** rpodolyaka1 has joined #tripleo | 17:13 | |
| *** jistr has quit IRC | 17:14 | |
| *** meena has joined #tripleo | 17:18 | |
| meena | I have a openstack env setup using TripleO devtest. I am trying to run tempest from undercloud against the overcloud non-compute. run_tests.sh complains about undefined client "cinderclient" Does it require all the clients installed explicitly? Is there any other way I can run tempest against TripleO? | 17:19 |
| bnemec | meena: I don't know how the tempest run_tests.sh works, but you might want to take a look at the tempest element: https://github.com/openstack/tripleo-image-elements/blob/master/elements/tempest/bin/run-tempest | 17:24 |
| bnemec | Just plain tempest out of the box isn't going to work against the overcloud. | 17:24 |
| dprince | meena: cinderclient should exist in the undercloud. It requires boot-stack which should always install all the clients (including cinderclient) | 17:25 |
| bnemec | When I was working on that element I was just including it in the overcloud control node and running it from there. It would be interesting to know if you can do the same thing with the undercloud. | 17:25 |
| *** vkozhukalov has joined #tripleo | 17:26 | |
| *** eghobo has quit IRC | 17:27 | |
| openstackgerrit | Jon-Paul Sullivan proposed a change to openstack/diskimage-builder: Standardise manifest creation and retrieval https://review.openstack.org/83848 | 17:28 |
| meena | dprince: you ran tempest from overcloud? | 17:28 |
| openstackgerrit | Jon-Paul Sullivan proposed a change to openstack/tripleo-image-elements: Standardise manifest creation and retrieval https://review.openstack.org/83850 | 17:28 |
| *** eghobo has joined #tripleo | 17:28 | |
| dprince | meena: no. I thought you were trying to run it from the undercloud (against the overcloud) | 17:29 |
| *** jcoufal has quit IRC | 17:29 | |
| bnemec | I ran it from the overcloud. In theory it should work from the undercloud too, but I don't know if anyone has done that. | 17:29 |
| *** geerdest has quit IRC | 17:29 | |
| dprince | bnemec: correct. I thought that was the question at hand here | 17:29 |
| *** geerdest has joined #tripleo | 17:30 | |
| *** lucasagomes is now known as lucas-afk | 17:30 | |
| bnemec | dprince: Right, and I was trying to say that I'm not sure run_tests.sh will work anyway. There's a bunch of stuff that run-tempest sets up in the tempest config that is required for it to work. | 17:32 |
| bnemec | So, to run tempest from the undercloud I'd suggest building an undercloud image with the tempest element, then trying to run it that way. | 17:33 |
| *** hashar has joined #tripleo | 17:34 | |
| bnemec | I think you also need the overcloud stackrc right now so you can get the appropriate credentials for tempest. | 17:34 |
| jp_at_hp | slagle: https://review.openstack.org/#/c/83848 and https://review.openstack.org/83850 should be good for review after your comments now - I've tested a run through locally and it passed clean... | 17:35 |
| *** rpodolyaka1 has quit IRC | 17:36 | |
| SpamapS | Hrm | 17:37 |
| *** rpodolyaka1 has joined #tripleo | 17:37 | |
| *** morganfainberg_Z is now known as morganfainberg | 17:38 | |
| Lotus907efi | does diskimage-builder pull "cloud" or qcow2 images from ubuntu and then massage them to work with tripleo? | 17:38 |
| Lotus907efi | or does it create images from scratch by pulling packages from a apt repo ? | 17:39 |
| SpamapS | Lotus907efi: it pulls the tar.gz's | 17:39 |
| SpamapS | Lotus907efi: http://cloud-images.ubuntu.com/saucy/current/saucy-server-cloudimg-i386-root.tar.gz | 17:40 |
| openstackgerrit | Trent Geerdes proposed a change to openstack/tripleo-incubator: Adding horizon element to undercloud https://review.openstack.org/84189 | 17:40 |
| *** martyntaylor has joined #tripleo | 17:40 | |
| *** akrivoka has joined #tripleo | 17:41 | |
| *** athomas has quit IRC | 17:41 | |
| *** e0ne has joined #tripleo | 17:41 | |
| meena | dprince / bnemec : I just need to run tempest. Does not matter even if it from overcloud. I just need to clone tempest on the overcloud, to run it? how do I install boot-stack on overcloud? | 17:41 |
| slagle | jp_at_hp: thanks, i'll check it out | 17:42 |
| dprince | meena: I think the recommendation is that you checkout the Tempest element, and follow the scripts there. | 17:42 |
| bnemec | Yep^ | 17:42 |
| bnemec | Just include the tempest element in the overcloud-control image and it will install tempest for you. | 17:43 |
| dprince | meena: as for boot-stack that recommendation was more for your question about running it from the undercloud. | 17:43 |
| SpamapS | meena: right, you could just build an overcloud-control image with the tempest element on it. | 17:43 |
| SpamapS | although.. | 17:43 |
| bnemec | Then follow the readme in https://github.com/openstack/tripleo-image-elements/tree/master/elements/tempest | 17:43 |
| dprince | meena: you should be able to run Tempest from several places though, including your seedhost machine so long as you have the right clients installed. For CI I think the plan is we are going to use the Jenkins slave to actually execute run_tests.sh | 17:44 |
| SpamapS | dprince: btw, I'm convinced that single-venv is necessary, even with the drawback that we lose individual venv-skew-problem-detection, after trying to work on a patch to make a --site-packages-from=/path/to/other/venv ... | 17:44 |
| dprince | SpamapS: \o/, cool | 17:44 |
| SpamapS | dprince: makes me sad, because I think we'll miss problems that people will find in the field.. but until we can reduce the disk footprint of all those venvs.. we're stuck. | 17:45 |
| dprince | SpamapS: I think someone (derekh perhaps) pointed out that we could have periodic jobs to cover the without case | 17:45 |
| Lotus907efi | SpamapS: thanks for the link to the tar,gz file | 17:45 |
| SpamapS | dprince: I'm not much of a fan of periodic jobs. They tend to snowball the moment the team gets busy. | 17:46 |
| SpamapS | Lotus907efi: you'll find copies of that in ~/.cache/image-create :) | 17:46 |
| Lotus907efi | is there anything "special" about the stuff in thaqt tar.gz file? It appears to be just a tar ball of a complete system | 17:46 |
| dprince | SpamapS: fair enough, just saying we could consider them as a stop gap to provide coverage | 17:46 |
| SpamapS | Lotus907efi: it is the root FS only. So we lay that down in tmpfs, chroot in, and work it over. Then we make a loopback, mkfs on that, and convert to qcow2. | 17:46 |
| Lotus907efi | ok | 17:47 |
| Lotus907efi | so could I build my own root fs tar ball to use? | 17:47 |
| SpamapS | dprince: I do think we can eventually have a hierarchy of venvs, but that is too far upstream for me to think I can land that _now_. | 17:47 |
| SpamapS | Lotus907efi: why would you do that? | 17:47 |
| Lotus907efi | and if so how would I get DIB to use that instead of trying to pull something from ubuntu? | 17:47 |
| SpamapS | Lotus907efi: is there something fundamentally wrong with the Ubuntu one? | 17:48 |
| *** rpodolyaka1 has quit IRC | 17:50 | |
| *** hashar has quit IRC | 17:51 | |
| *** jp_at_hp has quit IRC | 18:03 | |
| *** d0ugal_ has joined #tripleo | 18:06 | |
| *** d0ugal_ has quit IRC | 18:06 | |
| *** d0ugal_ has joined #tripleo | 18:07 | |
| *** jang1 has joined #tripleo | 18:07 | |
| *** d0ugal has quit IRC | 18:09 | |
| openstackgerrit | Dan Prince proposed a change to openstack/tripleo-heat-templates: Stop using notCompute in favor of controller https://review.openstack.org/84197 | 18:13 |
| *** bkero has quit IRC | 18:13 | |
| lifeless | morning | 18:16 |
| greghaynes | O/ | 18:17 |
| *** kevinbenton has joined #tripleo | 18:17 | |
| *** bkero has joined #tripleo | 18:18 | |
| jang1 | hey | 18:21 |
| lifeless | SpamapS: hierarchy of venvs? | 18:22 |
| lifeless | SpamapS: I'm thoroughly confused | 18:22 |
| lifeless | SpamapS: cam you look at the heat on undercloud for a sec? Its trying to create a novaclient with auth_url=None | 18:26 |
| *** rpodolyaka1 has joined #tripleo | 18:33 | |
| *** rpodolyaka1 has quit IRC | 18:39 | |
| *** bkero has quit IRC | 18:44 | |
| SpamapS | lifeless: so instead of --system-site-packages , we'd have a --site-packages-from= and then a path to a venv. | 18:47 |
| SpamapS | lifeless: allowing us to have a common venv. | 18:47 |
| SpamapS | lifeless: and then any venv specific ones would skew that. But .. it is a pipe dream. | 18:47 |
| lifeless | SpamapS: whats the point of that? | 18:48 |
| SpamapS | lifeless: as usual.. out of date api-paste.ini on heat | 18:48 |
| SpamapS | lifeless: 800MB smaller images | 18:48 |
| lifeless | SpamapS: a hard link optimiser would do the same | 18:48 |
| lifeless | SpamapS: and be much less entangled semantically | 18:49 |
| SpamapS | lifeless: Agreed.. I was thinking of going down that path next. | 18:49 |
| SpamapS | lifeless: also if we use wheels.. then we get the speed up already. | 18:49 |
| SpamapS | lifeless: api-paste.ini is out of date on cd-undercloud. Fixing | 18:49 |
| lifeless | SpamapS: so derekh said that the heat paste api had been changed in /etc rather than in the oac templatse | 18:49 |
| lifeless | so when we ran o-c-c --force --one it blew it away | 18:49 |
| *** bkero has joined #tripleo | 18:50 | |
| SpamapS | lifeless: fixed the oac templates too | 18:50 |
| *** dkehn_ has joined #tripleo | 18:50 | |
| SpamapS | lifeless: probably skipped that step on some day where I was less smart | 18:51 |
| lifeless | :) | 18:51 |
| lifeless | lets see now | 18:52 |
| lifeless | deleted testenv stack ok | 18:55 |
| lifeless | trying a deploy of a noo one | 18:55 |
| lifeless | SpamapS: rebooted the undercloud because networking was just 100% wedged | 18:56 |
| lifeless | SpamapS: last night my time | 18:56 |
| *** lucas-afk is now known as lucasagomes | 18:57 | |
| *** hashar has joined #tripleo | 19:01 | |
| *** rpodolyaka1 has joined #tripleo | 19:02 | |
| *** jcoufal has joined #tripleo | 19:04 | |
| *** rpodolyaka1 has quit IRC | 19:05 | |
| *** e0ne has quit IRC | 19:05 | |
| *** e0ne has joined #tripleo | 19:06 | |
| *** eguz has joined #tripleo | 19:06 | |
| *** rpodolyaka1 has joined #tripleo | 19:07 | |
| *** e0ne has quit IRC | 19:08 | |
| SpamapS | lifeless: looks better now yes? | 19:10 |
| *** eghobo has quit IRC | 19:10 | |
| jang1 | re hard link optimiser: tried that, my /opt shrank significantly. | 19:12 |
| lifeless | SpamapS: yes | 19:14 |
| lifeless | jang1: got a patch ? | 19:14 |
| jang1 | on my work box. If you can wait 12 hours, yeah. (Usually I turn up in the morning and stuff like that's been done overnight :-) ) | 19:16 |
| *** john-n-seattle has joined #tripleo | 19:16 | |
| *** rpodolyaka1 has quit IRC | 19:19 | |
| lifeless | SpamapS: now we're back to 'why isn't nova baremetal deploying things' | 19:19 |
| *** john-n-seattle has left #tripleo | 19:20 | |
| *** yassine has quit IRC | 19:24 | |
| SpamapS | lifeless: PXE problems? | 19:25 |
| lifeless | SpamapS: ENFI - info logging is so useless | 19:25 |
| lifeless | just upped to debug | 19:25 |
| lifeless | and restarted compute | 19:25 |
| *** yassine has joined #tripleo | 19:26 | |
| SpamapS | lifeless: I'd ILO onto the target machines too | 19:26 |
| lifeless | kicking the deploy again | 19:26 |
| Ng | lifeless: I did some manual testing on that wonky networking node, I can't get the thing to boot successfully, for love nor money, so I filed a jira and asked the DC Ops to run some diagnostics | 19:28 |
| lifeless | Ng: awesome | 19:28 |
| Ng | and then did some more poking at the other bad/suspectedbad machines and did some more jira-ing to get them looked at | 19:29 |
| lifeless | Ng: the reason I rebuilt was because its the same one failed a couple weeks back | 19:29 |
| lifeless | Ng: and thats not a coincidence | 19:29 |
| Ng | yeah, it's definitely wonkazoid | 19:29 |
| Ng | it won't even boot off whatever is left on its disk, it just throws an Illegal Opcode error and sulks | 19:30 |
| SpamapS | sullen angst ridden servers | 19:31 |
| * SpamapS goes to get lunch and ponder the universe.. bbiab | 19:31 | |
| Shrews | SpamapS: servers from the 90's :) | 19:31 |
| lifeless | SpamapS: ipmitool works both interactive and according to compute log | 19:33 |
| *** sballe has joined #tripleo | 19:34 | |
| lifeless | config is written correctly | 19:36 |
| lifeless | ipmi commands to request a PXE boot were executed | 19:43 |
| *** e0ne has joined #tripleo | 19:44 | |
| lifeless | the machine was sitting in 'attempting to boot from local disk' | 19:44 |
| lifeless | which is odd | 19:44 |
| lifeless | Ng: our images don't have boot blocks, so that isn't terribly surprising | 19:45 |
| *** andreaf has joined #tripleo | 19:47 | |
| *** lblanchard1 has quit IRC | 19:47 | |
| *** lblanchard1 has joined #tripleo | 19:48 | |
| slagle | lifeless: hi, do you think you could create the stable icehouse branches for tripleo-* this week? | 19:53 |
| slagle | or, i could volunteer to do releases this week, and do the stable branches at the same time i suppose | 19:53 |
| lifeless | slagle: +1 ;) | 19:53 |
| openstackgerrit | Ben Nemec proposed a change to openstack/tripleo-heat-templates: Allow setting the cinder iscsi-helper option https://review.openstack.org/78461 | 19:53 |
| slagle | lifeless: ok, i think i need permissions in gerrit though | 19:53 |
| lifeless | slagle: also we need to page in the thread about which things are being branched and which aren't :) | 19:53 |
| lifeless | slagle: yes, there's a team to add you to once you're up on all the docs and process | 19:54 |
| lifeless | tripleo-ptl | 19:55 |
| slagle | lifeless: are there any tripleo specific docs for the releases? | 19:55 |
| lifeless | slagle: rpodolyaka wrote some I think | 19:55 |
| slagle | or is it just pushing the signed tag | 19:55 |
| * slagle looks | 19:56 | |
| lifeless | slagle: list of projects (e.g. we don't erlease incubator) + deciding on the version + pushing the signed tag | 19:56 |
| lifeless | so, no reply from DHCP server | 19:56 |
| greghaynes | dprince: \O/ for name change, but theres going to be some cross-repo dependency awesomeness with getting that landed... | 19:56 |
| lifeless | restarting n-o-a | 19:56 |
| lifeless | greghaynes: which name change? | 19:57 |
| greghaynes | NotCompute -> Control in heat template | 19:57 |
| lifeless | dprince: btw I wanted to play with your persistent state patch, but cloud-down yesterday was all consuming | 19:57 |
| lifeless | dprince: today I shall, fingers crossed | 19:57 |
| *** lucasagomes is now known as lucas-dinner | 20:01 | |
| greghaynes | dprince: ugh, nvm, im thinking in the future ;) | 20:01 |
| * greghaynes remembers that his repo doesnt reflect current reality | 20:02 | |
| lifeless | greghaynes: what do you need to help ha move forward? reviews? testing? changes? | 20:02 |
| lifeless | bootpc not reacing the dhcp netns | 20:03 |
| greghaynes | well trying to figure out how to do the template scaling bit, which is hopefully going to be dependent on software-config yes? I was just trying to make a stack using that changeset | 20:03 |
| lifeless | restarting n-d-a | 20:03 |
| greghaynes | Also reviews on galera cluster patch from friday would be awesome | 20:03 |
| lifeless | to no apparent effect | 20:03 |
| greghaynes | https://review.openstack.org/#/c/83883/ | 20:04 |
| lifeless | ahha | 20:05 |
| lifeless | http://paste.openstack.org/show/74692/ | 20:05 |
| lifeless | fuuuuuuck | 20:05 |
| lifeless | WAT | 20:09 |
| lifeless | heat stack-delete testenv | 20:09 |
| lifeless | ERROR: You are not authorized to complete this action. | 20:10 |
| lifeless | SpamapS: please tell me that you did test your changes with an o-c-c --force --one :) | 20:10 |
| openstackgerrit | Gregory Haynes proposed a change to openstack/tripleo-incubator: Add OVERCLOUD_CONTROLSCALE support https://review.openstack.org/83296 | 20:13 |
| *** akuznetsov has quit IRC | 20:13 | |
| *** vkozhukalov has quit IRC | 20:14 | |
| *** rpodolyaka1 has joined #tripleo | 20:19 | |
| *** rwsu has joined #tripleo | 20:22 | |
| *** sballe has quit IRC | 20:24 | |
| *** rpodolyaka1 has quit IRC | 20:24 | |
| *** julim has quit IRC | 20:29 | |
| *** martyntaylor has left #tripleo | 20:33 | |
| *** martyntaylor has joined #tripleo | 20:35 | |
| *** eguz has quit IRC | 20:36 | |
| openstackgerrit | Ben Nemec proposed a change to openstack/diskimage-builder: set -e all the things https://review.openstack.org/83927 | 20:41 |
| openstackgerrit | Ben Nemec proposed a change to openstack/diskimage-builder: Make sure all scripts are set -e https://review.openstack.org/81637 | 20:41 |
| openstackgerrit | Ben Nemec proposed a change to openstack/diskimage-builder: dib-lint does not work with set -e https://review.openstack.org/83930 | 20:41 |
| openstackgerrit | Ben Nemec proposed a change to openstack/diskimage-builder: Check for set -o pipefail https://review.openstack.org/83929 | 20:41 |
| openstackgerrit | Ben Nemec proposed a change to openstack/diskimage-builder: Ensure scripts are set -u https://review.openstack.org/83928 | 20:41 |
| *** jtomasek has quit IRC | 20:48 | |
| *** e0ne has quit IRC | 20:50 | |
| *** e0ne has joined #tripleo | 20:51 | |
| *** e0ne has quit IRC | 20:55 | |
| *** jdob has quit IRC | 20:56 | |
| *** untriaged-bot has joined #tripleo | 21:00 | |
| untriaged-bot | Untriaged bugs so far: | 21:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1290488 | 21:00 |
| uvirtbot | Launchpad bug 1290488 in tripleo "Baremetal: Invalid credentials" [Undecided,Incomplete] | 21:00 |
| *** untriaged-bot has quit IRC | 21:00 | |
| *** eghobo has joined #tripleo | 21:03 | |
| SpamapS | lifeless: I fixed the templates and restarted heat manually. Still not entirely comfortable with "restart everything to enact one file change" | 21:05 |
| SpamapS | lifeless: but that's just a reflex, not a real problem :p | 21:05 |
| lifeless | SpamapS: yeah, you didn't fix policy.json in oac | 21:07 |
| lifeless | SpamapS: so when I triggered a restart, failed. | 21:07 |
| lifeless | SpamapS: can I make a small suggestion- please check occ works | 21:07 |
| lifeless | :) | 21:07 |
| *** dprince has quit IRC | 21:10 | |
| Lotus907efi | if I do "nova boot --flavor baremetal --image blah blah blah" it does it's thing and then returns a status message that among other things includes a password | 21:10 |
| Lotus907efi | how can I recover that password from nova? | 21:11 |
| jang1 | on the "restart everything" thing. Been thinking about that. Is there interest in os-svc-install (or similar) permitting the registration of a set of files or paths, and having some support to only restart a particular service if the contents of some of those files has changed? That'd be relatively simple to knock up. | 21:11 |
| Lotus907efi | I tried "nova baremetal-node-show ID" but that did not include enough info | 21:12 |
| jang1 | watching the number of restarts during an initial set of o-c-c runs just made me wonder whether it'd be a good idea | 21:12 |
| SpamapS | lifeless: I didn't fix policy.json ever | 21:13 |
| SpamapS | lifeless: so ??? :-P | 21:13 |
| SpamapS | jang1: I'm quite dubious that such a thing can be effective without complexity for python. | 21:14 |
| SpamapS | jang1: at least, not without stracing .. | 21:14 |
| *** pblaho has joined #tripleo | 21:15 | |
| lifeless | SpamapS: then derekh may have not fixed it correctly | 21:16 |
| jang1 | SpamapS: yeah; I was thinking more of config changes. With package-based installs, there's a trivial way to determine if you think a service needs a restart. Although it _would_ be possible to point what I've got in mind at a venv, I had in mind more of the sort of thing: "if this file changes, or that, or anything under /etc/nova ..." | 21:16 |
| lifeless | jang1: in principle we should only be going through things once | 21:16 |
| lifeless | jang1: are you seeing more than one ? | 21:16 |
| jang1 | well, there was that argument too :-) I was seeing multiple runs of o-c-c required to bring a host up, yeah. | 21:17 |
| lifeless | jang1: ok so thats a bug ;) | 21:17 |
| jang1 | eg, mysql started, something crapped out; mysql users created, something crapped out; mysql migrations applied | 21:17 |
| jang1 | I'd agree | 21:18 |
| jang1 | (that it's a bug) | 21:18 |
| jang1 | I think it's a big reason we're seeing such slow startup times for the devtest deployments | 21:18 |
| jang1 | I was thinking more of in general: heat updates just tweaking parameters, or the less-than-reimage updates. Although, I've no real feeling for how often those might happen in a tripleo environment. I just know how often they happen in the other environment I've had anything to do with | 21:19 |
| *** rpodolyaka1 has joined #tripleo | 21:20 | |
| jang1 | I've some baremetal work to do tomorrow but then I've a moderate sized list of things to go back over and figure out what's going wrong. | 21:20 |
| lifeless | whats a quota class vs a quota? | 21:21 |
| *** e0ne has joined #tripleo | 21:21 | |
| *** e0ne has quit IRC | 21:23 | |
| *** e0ne has joined #tripleo | 21:24 | |
| *** rpodolyaka1 has quit IRC | 21:24 | |
| SpamapS | jang1: I don't think package based installs make it any more trivial. Packages don't own all files that affect the running daemon. | 21:25 |
| *** lblanchard1 has quit IRC | 21:26 | |
| *** e0ne has quit IRC | 21:26 | |
| SpamapS | jang1: but.. I do think you could reasonably go the other way.. if you know only the config files for X changed, then only restart X. | 21:27 |
| SpamapS | jang1: it's just not as big of a win as the more broad version of that which tracks all files. | 21:27 |
| SpamapS | jang1: also regarding multiple occ runs being required, we put in some short circuits to make sure that we don't do that... like if occ sees that its own config file changed, it reexecs immediately, not after 30 seconds. | 21:28 |
| lifeless | http://paste.openstack.org/show/74700/ | 21:29 |
| lifeless | STABBBBB | 21:29 |
| lifeless | # Scheduler will always pick node with less ram first. When we will have | 21:30 |
| lifeless | # exact match, this will not be needed. | 21:30 |
| lifeless | ram_weight_multiplier=-1.0 | 21:30 |
| lifeless | ^ - entirely wrong | 21:30 |
| lifeless | on kvm instances | 21:30 |
| *** jang1 has quit IRC | 21:31 | |
| lifeless | some reviewers dropped the ball :( | 21:31 |
| *** lucas-dinner has quit IRC | 21:32 | |
| lifeless | filed https://bugs.launchpad.net/tripleo/+bug/1300458 | 21:32 |
| uvirtbot | Launchpad bug 1300458 in tripleo "incorrect ram weighting on overclouds" [Critical,Triaged] | 21:32 |
| *** jcoufal has quit IRC | 21:34 | |
| Lotus907efi | lifeless: I thought you said if I had to fix something with the pypi mirror, like adding a package by hand it would not remove it? | 21:34 |
| lifeless | Lotus907efi: thats correct, pypi-mirror is add-only | 21:35 |
| Lotus907efi | it removed my entries in the index.html | 21:36 |
| *** slagle has quit IRC | 21:36 | |
| *** andreaf has quit IRC | 21:38 | |
| *** slagle has joined #tripleo | 21:38 | |
| *** andreaf has joined #tripleo | 21:44 | |
| *** yassine has quit IRC | 21:56 | |
| *** pblaho has quit IRC | 21:58 | |
| *** e0ne has joined #tripleo | 21:59 | |
| *** andreaf has quit IRC | 22:00 | |
| *** tburnes has quit IRC | 22:01 | |
| *** rlandy has quit IRC | 22:03 | |
| tchaypo | morning | 22:04 |
| *** matty_dubs is now known as matty_dubs|gone | 22:05 | |
| Lotus907efi | if I use DIB to build an image and then boot that image to baremetal what does that image have in it for account credentials? | 22:07 |
| Lotus907efi | I have a precise baremetal node booted but I cannot login on the serial (ILO) console | 22:07 |
| *** andreaf has joined #tripleo | 22:08 | |
| tchaypo | my guess is that it would depend on what elements you used to build it | 22:08 |
| tchaypo | i think devtest uses the "stack-user" element to set up a default user | 22:09 |
| *** hashar has quit IRC | 22:09 | |
| Lotus907efi | ok, and where can I find any of the login creds for that default user? | 22:10 |
| Lotus907efi | I did use the stackuser element | 22:11 |
| Lotus907efi | when I built the image | 22:11 |
| Lotus907efi | nevermind | 22:12 |
| lifeless | stack and stack | 22:12 |
| Lotus907efi | I am in as "stack" | 22:12 |
| lifeless | the main vector for creds though is ec2 metadata which delivers a SSH key | 22:12 |
| lifeless | the one you chose when you did 'nova boot' | 22:12 |
| *** e0ne has quit IRC | 22:13 | |
| *** geerdest has quit IRC | 22:14 | |
| Lotus907efi | hmmm | 22:14 |
| Lotus907efi | the one I chose? | 22:15 |
| SpamapS | --key-name | 22:15 |
| Lotus907efi | oh | 22:15 |
| Lotus907efi | ok | 22:15 |
| SpamapS | usually "default" | 22:15 |
| *** geerdest has joined #tripleo | 22:15 | |
| Lotus907efi | yeah, and if it is "default" where does it get that ssh key from? | 22:16 |
| SpamapS | Lotus907efi: if you're doing this manually, you upload it with 'nova keypair-add' | 22:16 |
| Lotus907efi | ah | 22:16 |
| SpamapS | Lotus907efi: devtest automatically slurps your ~/.ssh/id_rsa.pub in as default | 22:16 |
| Lotus907efi | ok, so I probably missed that step then :) | 22:17 |
| SpamapS | Lotus907efi: Not much of a cloud user are you? ;) | 22:17 |
| Lotus907efi | no | 22:17 |
| Lotus907efi | all new to this | 22:17 |
| SpamapS | welcome to the new revolution! | 22:17 |
| SpamapS | It's like the old revolution | 22:17 |
| SpamapS | but on vms | 22:17 |
| Lotus907efi | (same as the old revolution) :) | 22:17 |
| * tchaypo learns from Lotus07efi's mistakes | 22:18 | |
| * tchaypo does not learn from grow@hp | 22:18 | |
| tchaypo | because the damn thing won't respond to me | 22:18 |
| greghaynes | tchaypo: is that the onboarding video stuff | 22:19 |
| tchaypo | yeah, i thought I'd completed it weeks ago but i got a nag email today telling me I'm overdue for another mandatory course | 22:20 |
| greghaynes | huh | 22:21 |
| greghaynes | I have yet to do but I also havent gotten any nag emails | 22:21 |
| clarkb | greghaynes: that greg hayne guy did them all for you | 22:22 |
| greghaynes | oh, greg hanes | 22:22 |
| greghaynes | that guy | 22:22 |
| tchaypo | progress! | 22:23 |
| tchaypo | I'm getting responses now | 22:23 |
| greghaynes | Sorry to hear | 22:23 |
| tchaypo | the response is that searching the lucene index failed | 22:23 |
| tchaypo | with java.lang.OutOfMemoryError | 22:24 |
| *** CaptTofu has joined #tripleo | 22:24 | |
| greghaynes | haha | 22:24 |
| greghaynes | SpamapS: With the software-config patch, am I supposed to be able to get a CREATE_COMPLETED when making overcloud? | 22:24 |
| greghaynes | my control vm seems to build and o-c-c succeed, just doesnt trigger the heat engine that alls well | 22:25 |
| SpamapS | greghaynes: there's a patch that has to land to t-i-e | 22:28 |
| lifeless | SpamapS: which one ? | 22:28 |
| SpamapS | https://review.openstack.org/#/c/83614/ | 22:28 |
| SpamapS | Oh look just needs another +2 | 22:29 |
| greghaynes | SpamapS: ok, I left you a comment for a left in pdb also | 22:29 |
| SpamapS | greghaynes: dohhhhh thanks | 22:29 |
| SpamapS | though I think that's because we don't actually use that bit now :) | 22:30 |
| greghaynes | What is the high level distinction between the Deployment and Config, like swift-deploy and switft-config | 22:30 |
| SpamapS | no thats just a validation.. hm | 22:30 |
| greghaynes | SpamapS: Yea, it didnt fail so figured it musnt be hitting it | 22:30 |
| SpamapS | yeah it's if your template is on crack :) | 22:30 |
| greghaynes | gah, I had totally seen that review too, just forgot | 22:32 |
| openstackgerrit | Steve Baker proposed a change to openstack/os-collect-config: Add collector for heat API resource metadata https://review.openstack.org/84269 | 22:33 |
| SpamapS | StevenK: \o/ ^^ | 22:34 |
| *** julim has joined #tripleo | 22:34 | |
| SpamapS | errrr | 22:34 |
| SpamapS | stevebaker: ^^^ | 22:34 |
| stevebaker | SpamapS: config is a re-usable chunk of configuration which allows inputs and outputs schema to be defined. A deployment makes a single config available to a single server, allowing server-specific input values to be defined and waiting IN_PROGRESS until output values are received | 22:38 |
| greghaynes | stevebaker: at me im assuming :) | 22:38 |
| SpamapS | :) | 22:38 |
| stevebaker | greghaynes: ^^ was meant for you | 22:38 |
| SpamapS | our scaling code is going to get unwieldy soon.. | 22:39 |
| greghaynes | aha, so the comments in SoftwareConfig class suddenly make a lot more sense | 22:39 |
| greghaynes | SpamapS: merge.py or other stuff? | 22:41 |
| lifeless | SpamapS: 'soon' | 22:41 |
| lifeless | SpamapS: 'going to' | 22:41 |
| lifeless | SpamapS: bwahahahahahahahahahahahaha | 22:41 |
| SpamapS | :) | 22:41 |
| openstackgerrit | Clint "SpamapS" Byrum proposed a change to openstack/tripleo-heat-templates: WIP: Switch overcloud to software-config https://review.openstack.org/81666 | 22:42 |
| SpamapS | hm actually we somehow steered clear of breaking undercloud | 22:44 |
| SpamapS | stevebaker: ugh, heatclient 0.2.8 vs. 0.2.3? | 22:46 |
| stevebaker | SpamapS: why 0.2.3? | 22:47 |
| openstackgerrit | A change was merged to openstack/os-collect-config: Updated from global requirements https://review.openstack.org/83483 | 22:48 |
| stevebaker | oh, requrements | 22:49 |
| SpamapS | stevebaker: yeah | 22:49 |
| stevebaker | it should work, maybe | 22:50 |
| SpamapS | stevebaker: seems like now would be a good time to update the global reqs anyway. :p | 22:50 |
| stevebaker | SpamapS: there is a requirements freeze :/ | 22:51 |
| openstackgerrit | Steve Baker proposed a change to openstack/os-collect-config: Add collector for heat API resource metadata https://review.openstack.org/84269 | 22:53 |
| *** julim has quit IRC | 23:00 | |
| *** CaptTofu has quit IRC | 23:00 | |
| mordred | stevebaker: the requirements freeze shouldn't block updating global reqs for python-heatclient - I believe an FFE would be quite justified there | 23:19 |
| mordred | unless I misunderstand the conversation | 23:19 |
| *** rpodolyaka1 has joined #tripleo | 23:20 | |
| SpamapS | It can wait I think | 23:20 |
| mordred | k | 23:20 |
| stevebaker | mordred: ok. I don't think os-collect-config needs a newer heatclient, but heat does. I'd like to do a heatclient release soon anyway | 23:20 |
| SpamapS | This is juno target stuff | 23:20 |
| mordred | I'm happy to champion an FFE if you need one | 23:20 |
| SpamapS | we'll be fine w/ cfn only for now | 23:20 |
| SpamapS | well my new templates seem to wokr here. | 23:21 |
| SpamapS | we'll see if CI agrees | 23:21 |
| SpamapS | if so I'll likely drop the WIP prefix.. bring on the softwareconfigs. :) | 23:21 |
| *** TravT has quit IRC | 23:21 | |
| *** yamahata has quit IRC | 23:21 | |
| stevebaker | \o/ | 23:21 |
| SpamapS | actually | 23:21 |
| SpamapS | CI will not pass until https://review.openstack.org/#/c/83614/ | 23:21 |
| SpamapS | :( | 23:21 |
| SpamapS | lifeless: ^^ blocking software configs | 23:22 |
| SpamapS | stevebaker: once I get that in.. I can make my custom resource plugin which makes a deployment and waits for the signal before rebuild or delete.... | 23:22 |
| SpamapS | well I have one but it is hacky .. will do it right | 23:22 |
| SpamapS | w/ like, tests and stuff :) | 23:22 |
| SpamapS | actually I could probably do it as soon as 83614 lands | 23:23 |
| SpamapS | but better ot have the templates and other bits working in similar fashion | 23:23 |
| lifeless | SpamapS: CI won't do jack until we get nodepool talking to the new cloud which is what we're discussing in openstack-infra atm | 23:23 |
| tchaypo | I've added a note to the TripleO wiki page asserting that people doing reviews should follow https://wiki.openstack.org/wiki/ReviewChecklist | 23:23 |
| *** rpodolyaka1 has quit IRC | 23:24 | |
| SpamapS | lifeless: OH .. thought RH region would be doing things still | 23:24 |
| lifeless | SpamapS: its not in nodepool yet | 23:25 |
| SpamapS | ah ok | 23:25 |
| SpamapS | well.. | 23:25 |
| * SpamapS -> kids | 23:25 | |
| SpamapS | when I get back.. :) | 23:25 |
| SpamapS | lifeless: I will try to poke at dhcp-all-interfaces again tonight | 23:26 |
| openstackgerrit | James Polley proposed a change to openstack/tripleo-incubator: Add some clarity to the first-time user experience https://review.openstack.org/83294 | 23:30 |
| *** geerdest has quit IRC | 23:32 | |
| openstackgerrit | James Polley proposed a change to openstack/tripleo-incubator: Add some clarity to the first-time user experience https://review.openstack.org/83294 | 23:34 |
| openstackgerrit | Loganathan Parthipan proposed a change to openstack/tripleo-incubator: Make libvirt volume pool configurable https://review.openstack.org/83409 | 23:47 |
| Lotus907efi | if I am doing the following command: DIB_RELEASE=precise DIB_APT_SOURCES=/etc/apt/sources.list_precise disk-image-create -a amd64 -o just_precise baremetal ubuntu stackuser --offline apt-sources use-ephemeral pypi use-ephemeral | 23:51 |
| Lotus907efi | is the precise cloud image tar.gz bit used at all? I.E. precise-server-cloudimg-amd64-root.tar.gz | 23:52 |
| Lotus907efi | if the bits in that tar.gz file are used when DIB is creating a new image, how is it used? | 23:55 |
| greghaynes | SpamapS: With the software-config change im getting an error when o-r-c runs os-apply-config --key db-password. Seems like db-passwords is only inside the deployments array... | 23:56 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!