| openstackgerrit | RedHat RDO CI proposed openstack/tripleo-heat-templates master: GATE CHECK for TripleO https://review.openstack.org/604298 | 00:00 |
|---|---|---|
| jidar | *ctrl-f NAT* 0 results... | 00:01 |
| jidar | where specifically does it say you need a NAT network? | 00:02 |
| *** tosky has quit IRC | 00:02 | |
| *** dsneddon has joined #tripleo | 00:03 | |
| *** ansmith has joined #tripleo | 00:09 | |
| *** dsneddon has quit IRC | 00:13 | |
| *** ssbarnea|rover has quit IRC | 00:13 | |
| mwhahaha | jidar: what are you talking about | 00:13 |
| mwhahaha | oh nm | 00:13 |
| mwhahaha | ska: yea it doesn't need a nat'd network unless your overcloud nodes need to communicate outside of the network environment (for example for ntp) | 00:14 |
| mwhahaha | ska: usually the pxe/provisioning network is nat'd or you masquerade via the undercloud | 00:15 |
| jidar | yea, I just setup the pxe network to use the undercloud as a management 'route' | 00:15 |
| ska | Can I use my br0 bride for that instead. Seems like a resonable choice. | 00:15 |
| mwhahaha | i don't know what br0 is in your env | 00:16 |
| ska | I mean instead of NAt'd, i use bridged | 00:16 |
| mwhahaha | there are some nic templates for using a single nic, but it requires you know what you re doing | 00:16 |
| jidar | are you asking if you can replace a interface with a bridge? no | 00:16 |
| *** ssbarnea|rover has joined #tripleo | 00:17 | |
| ska | No they get routed out of my system via the bridge.. Sorry for confusion. | 00:17 |
| jidar | I'm not even sure the undercloud installer will work that way mwhahaha | 00:17 |
| mwhahaha | oh i was talking about the overcloud | 00:17 |
| mwhahaha | undercloud you could probably get away with 1 nic but i've never tried | 00:18 |
| kplant | the undercloud works on a single nic | 00:18 |
| kplant | not fantastic practice to have the networks on a flat domain but it works | 00:18 |
| *** sdake has quit IRC | 00:19 | |
| *** kplant_ has joined #tripleo | 00:19 | |
| *** kplant has quit IRC | 00:19 | |
| *** kplant_ is now known as kplant | 00:19 | |
| mwhahaha | well it just means the provisionign network is routable | 00:19 |
| mwhahaha | you could still do multiple nics for the overcloud | 00:19 |
| mwhahaha | usually though the provisioning network isn't shared like that (hence the 2 nics) | 00:20 |
| kplant | on this subject, i was thinking of doing something funny with my overcloud nics... | 00:21 |
| * mwhahaha runs away | 00:22 | |
| kplant | :-) | 00:22 |
| kplant | each machine has two 40Gb nics, they will be lagged with lacp for 80Gb | 00:22 |
| kplant | i was thinking of pxe booting off one of them, obviously, and then moving the provisioning network into the lag as a sub interface | 00:22 |
| kplant | i figure that will work if the provisioning network vlan is native | 00:23 |
| kplant | and the lag is 'force-up' on the switch | 00:23 |
| mwhahaha | it might work | 00:23 |
| *** ansmith has quit IRC | 00:23 | |
| kplant | working with a qfx5110-32q i don't have any 10Gb or 1Gb ports to burn on pxe | 00:23 |
| mwhahaha | what no dlink? :D | 00:24 |
| openstackgerrit | Steve Baker proposed openstack/tripleo-common master: Break out tripleo-admin creation to its own role https://review.openstack.org/633827 | 00:25 |
| openstackgerrit | Steve Baker proposed openstack/tripleo-heat-templates master: Create tripleo-admin user on the undercloud https://review.openstack.org/633850 | 00:25 |
| *** saneax has quit IRC | 00:26 | |
| kplant | i could drill a hole in the side of one and put it in a rack | 00:26 |
| mwhahaha | i've always subscribed to having a separate management network infrastructure | 00:28 |
| mwhahaha | even if it's on crap gear | 00:28 |
| *** sdake has joined #tripleo | 00:28 | |
| kplant | the lom is on totally separate gear, so that's something i suppose | 00:28 |
| kplant | but yeah, not ideal | 00:29 |
| *** wolverineav has joined #tripleo | 00:29 | |
| jidar | we put it on the 1gig interfaces sometimes | 00:36 |
| jidar | the deploy goes slower that way, but it means better functional isolation | 00:36 |
| jidar | plus after the deployment, there's not much the interface needs to do | 00:36 |
| jidar | I'm having one goofy time right now trying to pin down why param_defaults doesn't get passed down properly or it's highly dependent which file it's included in | 00:37 |
| mwhahaha | ordering matters | 00:38 |
| mwhahaha | so it maybe overridden later | 00:38 |
| *** matbu has quit IRC | 00:39 | |
| *** dsneddon has joined #tripleo | 00:40 | |
| jidar | yea, but I'm getting some stuff like "params are defined but not used in plans" for stuff like Scheduler hints | 00:40 |
| mwhahaha | don't fret over that warning | 00:40 |
| jidar | I'm wondering if those warnings are bogus | 00:40 |
| mwhahaha | it's not completely accurate | 00:40 |
| jidar | but then I've got this osd_scenero defined but I'm still getting errors that it's not defined | 00:41 |
| kplant | did you typo osd_scenario? :> | 00:41 |
| mwhahaha | where so osd_scenario get defined | 00:41 |
| jidar | "msg": "[overcloud-ceph-all-2] Validation failed for variable: item[0]\n[overcloud-ceph-all-2] Reason: -> item[0] key did not match 'osd_scenario' (required item in schem | 00:42 |
| jidar | a is missing: osd_scenario)\n", | 00:42 |
| mwhahaha | what heat var do you set for that | 00:42 |
| jidar | er, so I had a ~/templates/ceph-custom-config.yaml, and it was included in my deploy command | 00:42 |
| mwhahaha | it looks to be part of CephAnsibleDisksConfig | 00:43 |
| mwhahaha | are you sure you aren't stomping on that later | 00:43 |
| mwhahaha | https://github.com/openstack/tripleo-heat-templates/blob/2ed0c19976cc2012d3482a6ffe89b1abb6556504/ci/environments/scenario004-standalone.yaml#L41-L45 | 00:43 |
| kplant | speaking of, have you been able to get lvm osd_scenario working with ceph-ansible 3.2? | 00:45 |
| kplant | i keep having to go back to filestore as i can't get lvm+bluestore to work, might be a layer 8 issue | 00:45 |
| *** ansmith has joined #tripleo | 00:46 | |
| jidar | mwhahaha: https://gist.github.com/08b5abc09e0ad2722eb25d41c4f352cd | 00:49 |
| jidar | it's not working for me this second kplant | 00:49 |
| *** kplant_ has joined #tripleo | 00:50 | |
| fultonj | jidar: that looks right to me | 00:50 |
| mwhahaha | jidar: is that last in your -e list? | 00:50 |
| kplant_ | i hate this vpn | 00:50 |
| *** kplant has quit IRC | 00:50 | |
| *** kplant_ is now known as kplant | 00:50 | |
| fultonj | jidar: have a look at your ceph-ansible inventory and confirm the scenario is defined there | 00:50 |
| jidar | mwhahaha: no, https://gist.github.com/017578c0f6acb4f25d9dde6c964b1be8 its in ceph-config.yaml | 00:51 |
| fultonj | to see the ceph-ansible inventory tell mistral to keep it around by setting the CephAnsiblePlaybookVerbosity > 0 as described in https://docs.openstack.org/tripleo-docs/latest/install/advanced_deployment/ceph_config.html#override-ansible-run-options | 00:51 |
| *** dsneddon has quit IRC | 00:51 | |
| mwhahaha | pretty sure ceph-config needs to be after ceph-ansible? | 00:51 |
| * mwhahaha defers to fultonj | 00:51 | |
| *** rlandy has quit IRC | 00:51 | |
| fultonj | jidar: swap lines 6 and 7 | 00:51 |
| jidar | ! | 00:51 |
| mwhahaha | last one wins :D | 00:51 |
| fultonj | i don't want defaults in ceph-ansible.yaml overriding what you set in ceph-config.yaml | 00:51 |
| jidar | wait, so it's the last one? | 00:52 |
| mwhahaha | same is probably true for your templates/* stuff | 00:52 |
| mwhahaha | yea | 00:52 |
| * jidar groans | 00:52 | |
| jidar | I thought it was the first one | 00:52 |
| mwhahaha | cmd line is parsed first to last | 00:52 |
| mwhahaha | the end overrides the rest | 00:52 |
| jidar | alright, 7pm here. Gona kick off this deploy and call it a night (I swear I will) thanks for the help guys, huge thank you and beer in the near future | 00:53 |
| fultonj | jidar: i have a hunch on your other issue | 00:53 |
| jidar | oh? | 00:53 |
| mwhahaha | move the scheduler hints to last? :D | 00:53 |
| fultonj | one sec, this bug might save you a little time regarding bluestore and the osd_scenario | 00:53 |
| mwhahaha | oh nm | 00:54 |
| * mwhahaha wanders off | 00:54 | |
| * fultonj gets bug | 00:54 | |
| fultonj | one sec | 00:54 |
| kplant | fultonj: is that directed at my question before i dropped? | 00:54 |
| fultonj | kplant: it might help you too | 00:54 |
| fultonj | are you both using loopback devices as disks for your OSDs? | 00:54 |
| kplant | no | 00:55 |
| fultonj | https://bugzilla.redhat.com/show_bug.cgi?id=1666822 | 00:55 |
| openstack | bugzilla.redhat.com bug 1666822 in Ceph-Volume "ceph-volume does not populate dictionary key rotational when passed a loopback device" [Unspecified,New] - Assigned to adeza | 00:55 |
| fultonj | whops, sorry guys maybe this doesn't apply | 00:56 |
| fultonj | in some dev envs i was using loopback and some docs suggest it | 00:57 |
| fultonj | to fake not having real disks | 00:57 |
| fultonj | bluestore doesn't handle it as per that bug ^ | 00:57 |
| fultonj | kplant: bluestore as what's it he docs works for me FWIW | 00:58 |
| fultonj | jidar: i guess you'll try the new order of arguments tomorrow | 00:58 |
| fultonj | g'night | 00:59 |
| kplant | fultonj: thanks i'll give it another go and check over my yaml | 00:59 |
| *** dsneddon has joined #tripleo | 01:00 | |
| jidar | yep, thanks ! | 01:03 |
| *** hamzy has joined #tripleo | 01:04 | |
| *** dsneddon has quit IRC | 01:04 | |
| *** ansmith has quit IRC | 01:11 | |
| *** sdake has quit IRC | 01:19 | |
| *** sdake has joined #tripleo | 01:23 | |
| openstackgerrit | Steve Baker proposed openstack/tripleo-heat-templates master: Create tripleo-admin user on the undercloud https://review.openstack.org/633850 | 01:36 |
| *** dsneddon has joined #tripleo | 01:37 | |
| *** wolverineav has quit IRC | 01:43 | |
| *** wolverineav has joined #tripleo | 01:44 | |
| *** tkajinam_ has joined #tripleo | 01:47 | |
| *** wolverineav has quit IRC | 01:48 | |
| *** tkajinam has quit IRC | 01:50 | |
| *** dsneddon has quit IRC | 01:57 | |
| *** dsneddon has joined #tripleo | 01:59 | |
| kplant | anyone know of the top of their head if you can do fencing with redfish instead of ipmi? | 01:59 |
| *** _fragatina has quit IRC | 02:05 | |
| *** _fragatina has joined #tripleo | 02:06 | |
| *** _fragatina has quit IRC | 02:07 | |
| *** dsneddon has quit IRC | 02:12 | |
| *** dsneddon has joined #tripleo | 02:12 | |
| *** wolverineav has joined #tripleo | 02:14 | |
| *** wolverineav has quit IRC | 02:18 | |
| *** dsneddon has quit IRC | 02:26 | |
| *** wolverineav has joined #tripleo | 02:30 | |
| *** wolverineav has quit IRC | 02:39 | |
| *** wolverineav has joined #tripleo | 02:40 | |
| *** mschuppert has quit IRC | 02:44 | |
| *** kplant has quit IRC | 02:53 | |
| *** dsneddon has joined #tripleo | 02:55 | |
| *** sdake has quit IRC | 03:00 | |
| *** wolverineav has quit IRC | 03:02 | |
| *** dsneddon has quit IRC | 03:03 | |
| *** dsneddon has joined #tripleo | 03:14 | |
| *** apetrich has quit IRC | 03:15 | |
| openstackgerrit | Rabi Mishra proposed openstack/tripleo-heat-templates master: WIP Check if mysql bootstrap is done for root password update https://review.openstack.org/633863 | 03:16 |
| *** ramishra has joined #tripleo | 03:23 | |
| *** dsneddon has quit IRC | 03:23 | |
| *** dsneddon has joined #tripleo | 03:30 | |
| *** skramaja has joined #tripleo | 03:32 | |
| *** dsneddon has quit IRC | 03:36 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/rocky: Autocreate CephAnsibleFetchDirectoryBackup https://review.openstack.org/633724 | 03:36 |
| openstackgerrit | Rabi Mishra proposed openstack/tripleo-heat-templates master: Add stop_grace_period for heat_engine container https://review.openstack.org/633864 | 03:37 |
| openstackgerrit | Merged openstack/python-tripleoclient stable/rocky: Handle tls endpoint for zaqar websocket client https://review.openstack.org/633024 | 03:43 |
| openstackgerrit | Merged openstack/paunch master: Raise exception on systemd failures https://review.openstack.org/623277 | 03:43 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/queens: In process-templates script write output files to provided dir when using base path https://review.openstack.org/633808 | 03:43 |
| *** ykarel|away has joined #tripleo | 03:43 | |
| *** wolverineav has joined #tripleo | 03:43 | |
| *** dsneddon has joined #tripleo | 03:44 | |
| *** wolverineav has quit IRC | 03:47 | |
| *** wolverineav has joined #tripleo | 03:47 | |
| *** dsneddon has quit IRC | 03:51 | |
| *** owalsh_ has joined #tripleo | 03:58 | |
| *** owalsh has quit IRC | 04:01 | |
| *** pbandark has quit IRC | 04:22 | |
| *** ayoung has quit IRC | 04:24 | |
| *** dsneddon has joined #tripleo | 04:28 | |
| *** dsneddon has quit IRC | 04:33 | |
| *** chkumar|out is now known as chandankumar | 04:38 | |
| *** agopi has quit IRC | 04:50 | |
| *** tzumainn has quit IRC | 05:00 | |
| *** aedc has quit IRC | 05:01 | |
| *** dsneddon has joined #tripleo | 05:05 | |
| *** dsneddon has quit IRC | 05:11 | |
| *** noama has joined #tripleo | 05:11 | |
| *** ykarel|away has quit IRC | 05:20 | |
| *** janki has joined #tripleo | 05:21 | |
| *** _fragatina has joined #tripleo | 05:23 | |
| *** _fragatina has quit IRC | 05:24 | |
| *** _fragatina has joined #tripleo | 05:24 | |
| jaosorior | EmilienM: need reviews? | 05:32 |
| *** ykarel|away has joined #tripleo | 05:32 | |
| *** ykarel|away is now known as ykarel | 05:33 | |
| EmilienM | jaosorior: no thanks! | 05:33 |
| jaosorior | ack | 05:34 |
| jaosorior | good morning everyone! | 05:34 |
| EmilienM | good night! | 05:34 |
| *** udesale has joined #tripleo | 05:34 | |
| *** dsneddon has joined #tripleo | 05:41 | |
| *** agopi has joined #tripleo | 05:45 | |
| *** udesale has quit IRC | 05:46 | |
| *** _fragatina has quit IRC | 05:46 | |
| *** _fragatina has joined #tripleo | 05:46 | |
| *** udesale has joined #tripleo | 05:48 | |
| *** udesale has quit IRC | 05:53 | |
| *** udesale has joined #tripleo | 05:54 | |
| *** psachin has joined #tripleo | 05:55 | |
| *** hakhande_ has joined #tripleo | 06:01 | |
| *** pdeore has joined #tripleo | 06:02 | |
| *** mmethot_ has joined #tripleo | 06:06 | |
| *** _mmethot_ has quit IRC | 06:06 | |
| *** mmethot_ has quit IRC | 06:12 | |
| *** mmethot has joined #tripleo | 06:14 | |
| *** nkinder has quit IRC | 06:28 | |
| *** agurenko has joined #tripleo | 06:35 | |
| *** quiquell|off is now known as quiquell | 06:39 | |
| *** yprokule has joined #tripleo | 06:42 | |
| *** pdeore has quit IRC | 06:44 | |
| *** ksambor has joined #tripleo | 06:56 | |
| *** waleedm has joined #tripleo | 06:58 | |
| Tengu | hello there :) | 06:59 |
| *** udesale has quit IRC | 07:02 | |
| *** matbu has joined #tripleo | 07:03 | |
| *** nkinder has joined #tripleo | 07:04 | |
| *** udesale has joined #tripleo | 07:05 | |
| *** udesale has quit IRC | 07:06 | |
| openstackgerrit | Cédric Jeanneret proposed openstack/puppet-tripleo master: Ensure we do not remove logs with very, very few traffic https://review.openstack.org/616945 | 07:07 |
| *** udesale has joined #tripleo | 07:08 | |
| *** udesale has quit IRC | 07:09 | |
| *** udesale has joined #tripleo | 07:09 | |
| *** radeks_ has joined #tripleo | 07:11 | |
| *** mandre|off is now known as mandre | 07:12 | |
| *** jfrancoa has joined #tripleo | 07:15 | |
| *** radeks_ has quit IRC | 07:17 | |
| *** saneax has joined #tripleo | 07:18 | |
| openstackgerrit | Kamil Sambor proposed openstack/tripleo-heat-templates master: Switch default neutron ML2 mechanism driver to OVN https://review.openstack.org/593056 | 07:22 |
| jaosorior | Tengu: the reasoning for that log purging is GDPR. So, we gotta make sure we stay compliant | 07:22 |
| *** radeks_ has joined #tripleo | 07:23 | |
| Tengu | jaosorior: I know. my change should keep the compliance. | 07:23 |
| Tengu | but still. logging management is terrible in tripleo -.-' | 07:23 |
| Tengu | brb, tea, breakfast | 07:23 |
| jaosorior | enjoy | 07:24 |
| *** kopecmartin|off is now known as kopecmartin | 07:26 | |
| *** cylopez has joined #tripleo | 07:27 | |
| *** numans is now known as numans_lunch | 07:29 | |
| *** ykarel is now known as ykarel|lunch | 07:30 | |
| Tengu | jaosorior: if we want a correct log management, all services should output to their stdout/stderr in the containers, and rsyslog should push the content in relevant files, and logrotate should run on the host and reload rsyslog or do whatever it needs. | 07:31 |
| jaosorior | Tengu: yeah... I tried doing that at some point | 07:32 |
| Tengu | jaosorior: that way, we would avoid the current situation: a log file is removed (too old) but the service in the container doesn't know it, and continue to lock the inode. | 07:32 |
| jaosorior | but didn't get enough traction | 07:32 |
| jaosorior | that's where the logging/ directory in the docker/services dir comes from | 07:32 |
| *** cfontain_ has quit IRC | 07:32 | |
| jaosorior | Tengu: for OSP16 we could finally push for having that as the default though | 07:32 |
| Tengu | jaosorior: I would say: it's mandatory now that all services are in containers | 07:33 |
| Tengu | also, this would alleviate all the work for monitoring | 07:33 |
| Tengu | letting rsyslog do its job is probably the best move we can do regarding logging. | 07:34 |
| jaosorior | Tengu: that would be good | 07:34 |
| Tengu | and we could drop a container.... | 07:35 |
| Tengu | the logrotate one would be meaningless. | 07:35 |
| jaosorior | Tengu: which container could we drop? | 07:35 |
| Tengu | all the best | 07:35 |
| jaosorior | ah | 07:35 |
| jaosorior | that | 07:35 |
| jaosorior | yeah | 07:35 |
| jaosorior | well, some containers would need an rsyslog container as a helper | 07:35 |
| jaosorior | to actually output to stdout | 07:35 |
| jaosorior | actually... we used to have an rsyslog container | 07:36 |
| jaosorior | and it got dropped | 07:36 |
| jaosorior | cause, again, that initiative never got traction | 07:36 |
| Tengu | errr | 07:36 |
| Tengu | why a rsyslog container when the host has a working rsyslog?? | 07:36 |
| Tengu | by default | 07:36 |
| jaosorior | Tengu: just as a side-car container that grabs the logs from the container, and pipes them to stdout | 07:37 |
| jaosorior | e.g. haproxy | 07:37 |
| Tengu | nope | 07:37 |
| Tengu | haproxy already writes in /dev/log | 07:37 |
| jaosorior | which is not a good practice either. Containers shouldn't need to mount /dev/log | 07:37 |
| jaosorior | we loose all the metadata from the container | 07:38 |
| Tengu | hence already has its logs managed by the host rsyslog | 07:38 |
| jaosorior | when containers logs to stdout, there's a bunch of metadata added to those logs | 07:38 |
| jaosorior | that gets lost when you do that | 07:38 |
| * Tengu trying to remember how logs were managed at his previous work for haproxy | 07:38 | |
| Tengu | hmm. I think we actually got some side-car for the output indeed. | 07:39 |
| Tengu | but we pushed directly into kafka back then | 07:39 |
| jaosorior | Tengu: it doesn't need to be an rsyslog sidecar; could be anything else | 07:39 |
| jaosorior | I don't mind really | 07:39 |
| jaosorior | as long as we stop bind-mounting /dev/log | 07:39 |
| *** florianf has joined #tripleo | 07:40 | |
| Tengu | well, it will need to expose something like /dev/log to haproxy anyway | 07:40 |
| jaosorior | Tengu: that socket is configurable AFAIK | 07:40 |
| Tengu | location? yep | 07:41 |
| Tengu | hence the "something like" | 07:41 |
| Tengu | it can as well be a network socket iirc | 07:41 |
| Tengu | so we might even NOT need to bind-mount the /dev/log | 07:41 |
| Tengu | we could configure rsyslog to listen on 127.0.0.1 - as containers are using "net=host" anyway. | 07:41 |
| Tengu | yeah, that's how we did back then at c2c. | 07:42 |
| jaosorior | Tengu: check docker/services/logging/stdout/haproxy.yaml | 07:42 |
| Tengu | so bind-mounts inter-container | 07:44 |
| jaosorior | Tengu: right | 07:44 |
| Tengu | oh by the way. podman now has a lame support for named-volumes! | 07:44 |
| jaosorior | oho | 07:44 |
| jaosorior | since when? | 07:44 |
| Tengu | I stumbled on that this weekend while trying to configure minio | 07:44 |
| Tengu | since 1.0 I think? | 07:44 |
| Tengu | BUT | 07:44 |
| Tengu | it still doesn't create the host directory. | 07:44 |
| Tengu | so basically, we need to create /var/lib/containers/storage/mounts/foo then we can podman run -v foo:/bar/baz .... | 07:45 |
| Tengu | unless podman volume does it - I didn't check that, tbh | 07:45 |
| Tengu | I was to surprised by the error I got when I tried to run podman -v /var/lib/containers/storage/mounts/foo:/foo :D | 07:45 |
| Tengu | (yeah, of course, the thing is not compatible with full path in that case) | 07:46 |
| jaosorior | anyway, anyway | 07:46 |
| jaosorior | I would be happy if we try to go again that path (do logging of containers to stdout) | 07:46 |
| Tengu | jaosorior: I can try to get involved into it for future sprint | 07:47 |
| jaosorior | Tengu: that would also help us flatten the heat stacks | 07:48 |
| Tengu | that would help convince my manager (and get some traction from DF :D) | 07:48 |
| Tengu | brb, cat cuddling | 07:48 |
| Tengu | :] | 07:50 |
| Tengu | the tiny beast is happy and purring. so. yeah. | 07:51 |
| *** gkadam has joined #tripleo | 07:51 | |
| jaosorior | lol | 07:51 |
| Tengu | jaosorior: we might want to create some spec for that? | 07:51 |
| Tengu | in order to provide the advantages, some fact listing and the like? | 07:51 |
| jaosorior | Tengu: https://specs.openstack.org/openstack/tripleo-specs/specs/queens/logging-stdout.html | 07:52 |
| *** ccamacho has joined #tripleo | 07:52 | |
| jaosorior | gotta revibe that | 07:52 |
| jaosorior | Tengu: https://blueprints.launchpad.net/tripleo/+spec/logging-stdout-rsyslog | 07:52 |
| Tengu | cool | 07:52 |
| Tengu | jaosorior: oh, and as we manage the containers pod via systemd, I wouldn't be surprised if we could run "jounalct -f -u <container-unit-name>" now | 07:53 |
| Tengu | that would be just awesome. | 07:53 |
| jaosorior | Tengu: that would actually already be available | 07:53 |
| jaosorior | even without podman | 07:53 |
| jaosorior | docker's default logging driver in CentOS is journald | 07:54 |
| Tengu | ah, via the CONTAINER_NAME=foo var passed to journalctl? | 07:54 |
| jaosorior | yeah | 07:54 |
| Tengu | cool | 07:54 |
| Tengu | that's lovely. | 07:54 |
| jaosorior | and if you expand (I forgot the flag) | 07:54 |
| *** mschuppert has joined #tripleo | 07:54 | |
| jaosorior | it gives you the container ID that the log originated from | 07:54 |
| jaosorior | the name | 07:54 |
| jaosorior | and other nice metadata | 07:54 |
| jaosorior | that's exactly the stuff I wanna keep for the containers, and the reason that I introduced the rsyslog sidecar | 07:54 |
| Tengu | makes sense. | 07:55 |
| jaosorior | once you forward the logs with rsyslog, you can send that metadata as well towards... wherever you decide to send it in json format | 07:55 |
| Tengu | I think we actually injected stuff in the logging, back then. but I don't remember, it's been almost a year... | 07:55 |
| Tengu | and yep, we did send logs into a kafka cluster, for elasticsearch and other treats. | 07:55 |
| Tengu | it was pretty awesome | 07:56 |
| jaosorior | anyway | 07:56 |
| jaosorior | lets revive that spec | 07:57 |
| Tengu | yess | 07:57 |
| jaosorior | and try to get buy-in from folks | 07:57 |
| jaosorior | there is already a bunch of work done there | 07:57 |
| jaosorior | so, it would be a matter of flattening some of the heat stacks | 07:57 |
| jaosorior | and removing the default file logging | 07:57 |
| Tengu | we will basically need to update some stuff, and flattening | 07:57 |
| jaosorior | and then... well... completing the work | 07:57 |
| Tengu | cool | 07:57 |
| Tengu | care to send a couple of emails about that? you get more traction/attention than me ^^ | 07:58 |
| jaosorior | haha do I? | 07:58 |
| *** slaweq has joined #tripleo | 07:58 | |
| Tengu | you're older in the place :D | 07:58 |
| jaosorior | I can | 07:58 |
| jaosorior | I'll send a mail to both the upstream mailing list | 07:58 |
| Tengu | perfect | 07:59 |
| *** jtomasek has joined #tripleo | 08:00 | |
| openstackgerrit | RedHat RDO CI proposed openstack/tripleo-heat-templates stable/queens: GATE CHECK for TripleO https://review.openstack.org/567224 | 08:00 |
| openstackgerrit | RedHat RDO CI proposed openstack/tripleo-heat-templates stable/rocky: GATE CHECK for TripleO https://review.openstack.org/604293 | 08:00 |
| *** florianf has quit IRC | 08:01 | |
| *** florianf has joined #tripleo | 08:03 | |
| *** agurenko has quit IRC | 08:04 | |
| *** agurenko has joined #tripleo | 08:06 | |
| *** waleedm has quit IRC | 08:10 | |
| *** pcaruana has joined #tripleo | 08:11 | |
| *** tkajinam_ has quit IRC | 08:15 | |
| *** agurenko has quit IRC | 08:16 | |
| *** ykarel|lunch is now known as ykarel | 08:22 | |
| openstackgerrit | waleed mousa proposed openstack/os-net-config master: Configure switchdev mode in SR-IOV in os-net-config https://review.openstack.org/633885 | 08:24 |
| *** apetrich has joined #tripleo | 08:25 | |
| *** hberaud|gone is now known as hberaud | 08:27 | |
| *** bnemec has joined #tripleo | 08:31 | |
| *** aedc has joined #tripleo | 08:32 | |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-quickstart-extras master: Remove usage of target_upgrade_version. https://review.openstack.org/607525 | 08:32 |
| openstackgerrit | Raviv proposed openstack/tripleo-upgrade stable/queens: Add cli_opts_params.yaml to update prepare/converge. https://review.openstack.org/633886 | 08:37 |
| *** xek has joined #tripleo | 08:38 | |
| openstackgerrit | Chandan Kumar proposed openstack-infra/tripleo-ci master: Run tempest using os_tempest role in standalone job https://review.openstack.org/627500 | 08:39 |
| *** numans_lunch is now known as numans | 08:41 | |
| *** gkadam has quit IRC | 08:42 | |
| *** avivgt has joined #tripleo | 08:48 | |
| openstackgerrit | Lee Yarwood proposed openstack/tripleo-heat-templates master: placement: Extract the service from Nova https://review.openstack.org/630644 | 08:52 |
| *** tosky has joined #tripleo | 08:55 | |
| *** sanjayu_ has joined #tripleo | 08:56 | |
| *** jpich has joined #tripleo | 08:56 | |
| *** sanjayu_ has quit IRC | 08:57 | |
| *** jpena|off is now known as jpena | 08:57 | |
| openstackgerrit | Quique Llorente proposed openstack/tripleo-quickstart master: Dump virt-resize result to a file https://review.openstack.org/633444 | 09:03 |
| openstackgerrit | Quique Llorente proposed openstack/tripleo-quickstart master: Dump virt-resize result to a file https://review.openstack.org/633444 | 09:07 |
| noama | hi | 09:09 |
| noama | im trying to deploy overcloud with master branch getting the following issue | 09:10 |
| noama | https://www.irccloud.com/pastebin/QMpzhQxq/ | 09:10 |
| openstackgerrit | Michele Baldessari proposed openstack/puppet-pacemaker master: WIP Initial pcs 0.10 support https://review.openstack.org/607873 | 09:10 |
| openstackgerrit | Michele Baldessari proposed openstack/puppet-pacemaker master: WIP Initial support for properties managed by pcs 0.10 https://review.openstack.org/626005 | 09:10 |
| noama | can someone help me understand what causing the issue above | 09:11 |
| ramishra | hi guys, anyone looked at the timeouts happening for many jobs. It seems to be stuck in host configuration step2 for undercloud install http://logs.openstack.org/98/604298/204/check/tripleo-ci-centos-7-undercloud-containers/5a92241/logs/undercloud/home/zuul/install-undercloud.log.txt.gz | 09:11 |
| ramishra | looks to me possibly something to do with firewall rule changes we did recently, Tengu^^^ do you have an idea? | 09:12 |
| ykarel | arxcruz|ruck, ssbarnea|rover ^^ | 09:13 |
| Tengu | ramishra: have some doubts. firewall rules are pushed at step 3 or 4 | 09:20 |
| *** gfidente has joined #tripleo | 09:20 | |
| Tengu | ramishra: and if you could point a line or something in the log... ? | 09:20 |
| ramishra | http://logs.openstack.org/98/604298/204/check/tripleo-ci-centos-7-containers-multinode/008eb3d/logs/undercloud/home/zuul/undercloud_install.log.txt.gz#_2019-01-30_00_45_11 | 09:21 |
| ramishra | Tengu: this the step for a successful run | 09:21 |
| ramishra | and it seems to be doing Firewall changes | 09:22 |
| *** bogdando has joined #tripleo | 09:22 | |
| Tengu | hmmm nope. | 09:22 |
| Tengu | saving rules doesn't change the firewall. | 09:22 |
| Tengu | but here, maybe, there's something: http://logs.openstack.org/98/604298/204/check/tripleo-ci-centos-7-containers-multinode/008eb3d/logs/undercloud/home/zuul/undercloud_install.log.txt.gz#_2019-01-30_00_55_16 | 09:22 |
| ramishra | Tengu: I'm just guessing, but it seems to be stuck in that step for all jobs timing out | 09:22 |
| ramishra | there may be;) | 09:23 |
| *** pbandark has joined #tripleo | 09:23 | |
| Tengu | weird part: I didn't see this behavior at all with my tests. And I ran many of them. | 09:23 |
| Tengu | ""Notice: Scope(Class[Tripleo::Firewall::Post]): At this stage, all network traffic is blocked."," | 09:23 |
| *** mmethot has quit IRC | 09:23 | |
| *** mmethot has joined #tripleo | 09:24 | |
| Tengu | ramishra: ok, so, actually, the rules are added in the right order, later in step 1 apparently. | 09:25 |
| Tengu | but the "lockout" is added *after* the opening of SSH and others. | 09:25 |
| *** saneax has quit IRC | 09:25 | |
| openstackgerrit | Quique Llorente proposed openstack/tripleo-quickstart master: Dump virt-resize result to a file https://review.openstack.org/633444 | 09:25 |
| Tengu | and, in the log you first pointed: http://logs.openstack.org/98/604298/204/check/tripleo-ci-centos-7-undercloud-containers/5a92241/logs/undercloud/home/zuul/install-undercloud.log.txt.gz#_2019-01-30_00_54_40_085 | 09:25 |
| *** wolverineav has quit IRC | 09:26 | |
| Tengu | ramishra: but there's a really WEIRD part: no mention of rule 003!! That 003 opens ssh. | 09:26 |
| Tengu | I'm wondering.... wait. | 09:27 |
| *** saneax has joined #tripleo | 09:27 | |
| ramishra | Tengu: Is this the patch responsible? https://github.com/openstack/puppet-tripleo/commit/f25c27aa2c6eff327d612d163c1758b59618d6ed ? | 09:28 |
| Tengu | yes, but no. | 09:28 |
| Tengu | there's another issue. | 09:28 |
| Tengu | my patch actually showed that issue :). | 09:28 |
| ramishra | Tengu: May be we can revert these for the time being to unblock. Which other patch you mean? | 09:29 |
| Tengu | https://review.openstack.org/#/c/631784/ | 09:29 |
| Tengu | puppet-tripleo wants tags. | 09:29 |
| Tengu | lemme add it. | 09:30 |
| openstackgerrit | Cédric Jeanneret proposed openstack/tripleo-heat-templates master: Add missing tag for puppet-tripleo https://review.openstack.org/633897 | 09:30 |
| Tengu | ramishra: -^^ should help a lot I guess. | 09:31 |
| Tengu | although, as said: I didn't see this behavior while testing my other patch... | 09:32 |
| openstackgerrit | Quique Llorente proposed openstack/tripleo-quickstart master: Dump virt-resize result to a file https://review.openstack.org/633444 | 09:32 |
| *** dciabrin has joined #tripleo | 09:34 | |
| Tengu | lemme push that patch on my local test-bench as well. | 09:35 |
| *** wolverineav has joined #tripleo | 09:35 | |
| bogdando | o/ please merge https://review.openstack.org/#/c/633484/ | 09:36 |
| *** ramishra has quit IRC | 09:40 | |
| *** ramishra has joined #tripleo | 09:40 | |
| ramishra | Tengu: raised https://bugs.launchpad.net/tripleo/+bug/1813900 to track | 09:41 |
| openstack | Launchpad bug 1813900 in tripleo "tripleo-ci-centos-7-undercloud-containers timing out for host configuration for step 2" [Critical,New] | 09:41 |
| *** paramite has joined #tripleo | 09:44 | |
| Tengu | ramishra: commented with some details and explanation. pointing to my patch doesn't help, and is just wrong. | 09:47 |
| openstackgerrit | Cédric Jeanneret proposed openstack/tripleo-heat-templates master: Add missing tag for puppet-tripleo https://review.openstack.org/633897 | 09:49 |
| Tengu | updated commit message to link it to the LP. | 09:49 |
| ramishra | Tengu: Something changed for it to break, no? I just guessed that it's something to do with firewall changes. Anyway, what matters is we fix it:) | 09:50 |
| Tengu | ramishra: root cause is: we had unmanaged rules in the firewall and it worked "by luck" :D. anyway. testing my patch on the lab (will be faster than CI), and as soon as zuul is happy, we can merge it. | 09:52 |
| *** shyamb has joined #tripleo | 09:58 | |
| *** derekh has joined #tripleo | 09:58 | |
| openstackgerrit | Sagi Shnaidman proposed openstack/openstack-virtual-baremetal master: WIP: Import auth with try/except https://review.openstack.org/633681 | 10:00 |
| openstackgerrit | Sagi Shnaidman proposed openstack/openstack-virtual-baremetal master: Import auth with try/except https://review.openstack.org/633681 | 10:01 |
| openstackgerrit | Quique Llorente proposed openstack/tripleo-quickstart master: Dump virt-resize result to a file https://review.openstack.org/633444 | 10:02 |
| openstackgerrit | Michele Baldessari proposed openstack/puppet-pacemaker master: WIP Initial pcs 0.10 support https://review.openstack.org/607873 | 10:03 |
| openstackgerrit | Michele Baldessari proposed openstack/puppet-pacemaker master: WIP Initial support for properties managed by pcs 0.10 https://review.openstack.org/626005 | 10:03 |
| *** owalsh_ is now known as owalsh | 10:03 | |
| Tengu | ramishra: some more digging: I can abandon my patch, tag is set directly in tripleo::firewall::rule. and some more digging in the log you first pointed shows the ssh opening being added, BEFORE the lockout. | 10:06 |
| Tengu | ramishra: so I think it's another issue, not related to the firewall changes. | 10:06 |
| jaosorior | the undercloud_containers job seems to be timing out | 10:07 |
| *** holser_ has joined #tripleo | 10:09 | |
| Tengu | jaosorior: https://bugs.launchpad.net/tripleo/+bug/1813900 | 10:09 |
| openstack | Launchpad bug 1813900 in tripleo "tripleo-ci-centos-7-undercloud-containers timing out in host configuration for step 2" [Critical,In progress] - Assigned to Cédric Jeanneret (cjeanner) | 10:09 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates master: Ensure all upgrade tasks depend on a step. https://review.openstack.org/628154 | 10:09 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates master: WIP: Split upgrade_steps_playbook into different plays. https://review.openstack.org/625890 | 10:09 |
| Tengu | already working on it, trying to figure out what's going on. Apparently it's NOT related to the firewall. | 10:10 |
| openstackgerrit | Sagi Shnaidman proposed openstack/openstack-virtual-baremetal stable/1.0: DNM: test ci for 1.0 https://review.openstack.org/633905 | 10:10 |
| *** shyamb has quit IRC | 10:13 | |
| Tengu | ramishra: http://logs.openstack.org/98/604298/204/check/tripleo-ci-centos-7-undercloud-containers/5a92241/logs/undercloud/var/log/extra/denials.txt.gz check the amount of denied related to haproxy connections near the end.... | 10:13 |
| *** shyamb has joined #tripleo | 10:13 | |
| Tengu | that might be the issue. | 10:13 |
| Tengu | hm. permissive=1. damn. | 10:14 |
| *** dr_gogeta86 has quit IRC | 10:15 | |
| *** suuuper has joined #tripleo | 10:17 | |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates stable/rocky: DO NOT MERGE testing for rocky standalones https://review.openstack.org/631492 | 10:17 |
| *** sshnaidm has quit IRC | 10:19 | |
| *** dr_gogeta86 has joined #tripleo | 10:20 | |
| openstackgerrit | Michele Baldessari proposed openstack/puppet-pacemaker master: WIP Initial pcs 0.10 support https://review.openstack.org/607873 | 10:21 |
| openstackgerrit | Michele Baldessari proposed openstack/puppet-pacemaker master: WIP Initial support for properties managed by pcs 0.10 https://review.openstack.org/626005 | 10:21 |
| arxcruz|ruck | gouthamr: around? any idea about this http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset019-master/8d38ff9/logs/undercloud/home/zuul/tempest.log.txt.gz ? | 10:22 |
| arxcruz|ruck | vkmc: ^ | 10:23 |
| arxcruz|ruck | manila tests failing | 10:23 |
| *** dtantsur|afk is now known as dtantsur | 10:25 | |
| Tengu | oh. ohoo | 10:26 |
| *** ramishra has quit IRC | 10:26 | |
| *** sshnaidm has joined #tripleo | 10:26 | |
| *** derekh has quit IRC | 10:29 | |
| *** paramite has quit IRC | 10:33 | |
| openstackgerrit | Quique Llorente proposed openstack/tripleo-quickstart master: Dump virt-resize result to a file https://review.openstack.org/633444 | 10:35 |
| *** boazel_ has joined #tripleo | 10:38 | |
| *** ramishra has joined #tripleo | 10:39 | |
| *** boazel has quit IRC | 10:41 | |
| *** paramite has joined #tripleo | 10:41 | |
| openstackgerrit | Raviv proposed openstack/tripleo-upgrade stable/queens: Add cli_opts_params.yaml to update prepare/converge. https://review.openstack.org/633911 | 10:43 |
| *** paramite has quit IRC | 10:45 | |
| *** paramite has joined #tripleo | 10:45 | |
| Tengu | ramishra: I'm digging in the instance logs - I don't see anything regarding potential failure with that step2 :/. All I can say: not related to the firewall. | 10:50 |
| Tengu | ramishra: http://logs.openstack.org/98/604298/204/check/tripleo-ci-centos-7-undercloud-containers/5a92241/logs/undercloud/var/log/journal.txt.gz#_Jan_30_00_59_47 we can see haproxy being started, but it doesn't have any backend apparently. | 10:51 |
| *** derekh has joined #tripleo | 10:51 | |
| Tengu | ramishra: ah, you also have seen them :). But I *think* it's normal, at least we can see those lines when we deploy an undercloud. haproxy configuration is pushed before services are actually started. | 10:52 |
| jaosorior | Tengu: that is the case | 10:53 |
| jaosorior | haproxy starts and it keeps reporting the servers are down... until step 3 and 4 when the services are actually started | 10:53 |
| Tengu | jaosorior: ah, that's what I've commented in the LP :). | 10:54 |
| Tengu | so my memory was right, step 3. | 10:54 |
| Tengu | nice thing: haproxy doesn't obsess on the reporting. | 10:54 |
| *** salmankhan has joined #tripleo | 10:55 | |
| openstackgerrit | Lukas Bezdicka proposed openstack/puppet-tripleo master: WIP Enable keystone authtoken caching on all the services https://review.openstack.org/633701 | 10:56 |
| openstackgerrit | Lukas Bezdicka proposed openstack/puppet-tripleo master: WIP Introduce keystone cache by default https://review.openstack.org/633915 | 10:56 |
| openstackgerrit | Quique Llorente proposed openstack/tripleo-quickstart master: Use force_tcg by default https://review.openstack.org/633444 | 10:57 |
| *** wolverineav has quit IRC | 10:59 | |
| Tengu | ramishra, jaosorior : hmm, as it seems to stuck on step 2 for the *host configuration*, I don't think any network is involved: it's on the undercloud, and we run that with "puppet apply <options>". Connections on localhost are all allowed according to the firewall: "-A INPUT -i lo -m state --state NEW -m comment --comment "002 accept all to lo interface ipv4" -j ACCEPT" | 11:00 |
| Tengu | also we have, before that, the RELATED,ESTABLISHED. ansible uses persistent connection/tunnels, so we shouldn't get any trouble. | 11:00 |
| *** paramite has quit IRC | 11:01 | |
| Tengu | and, in any cases, the following allows ssh anyway: "-A INPUT -p tcp -m multiport --dports 22 -m state --state NEW -m comment --comment "003 accept ssh from all ipv4" -j ACCEPT" | 11:01 |
| *** paramite has joined #tripleo | 11:01 | |
| Tengu | so, I really don't see any issue with the firewall and/or network :/. | 11:01 |
| Tengu | wait... we don't have ipv6 output | 11:02 |
| Tengu | bbl, lunch. | 11:03 |
| openstackgerrit | Michele Baldessari proposed openstack/puppet-pacemaker master: WIP Initial pcs 0.10 support https://review.openstack.org/607873 | 11:07 |
| openstackgerrit | Michele Baldessari proposed openstack/puppet-pacemaker master: WIP Initial support for properties managed by pcs 0.10 https://review.openstack.org/626005 | 11:07 |
| *** salmankhan has quit IRC | 11:10 | |
| *** shyamb has quit IRC | 11:15 | |
| *** pbandark has quit IRC | 11:18 | |
| panda | derekh: scenario012 | 11:18 |
| *** paramite has quit IRC | 11:22 | |
| *** salmankhan has joined #tripleo | 11:22 | |
| *** paramite has joined #tripleo | 11:22 | |
| *** holser_ is now known as holser|lunch | 11:23 | |
| panda | derekh: let's talk about it | 11:23 |
| derekh | panda: sure | 11:25 |
| derekh | panda: I hadn't noticed you +W one of the patches, thanks | 11:25 |
| derekh | panda: there is a depended one, also https://review.openstack.org/#/c/509728/ | 11:26 |
| *** udesale has quit IRC | 11:26 | |
| *** dsneddon has quit IRC | 11:27 | |
| derekh | panda: will update the description now | 11:27 |
| panda | derekh: we are trying to identify scenarios that can be converted to standalone, it doesn't seem to be ready for the conversion, but I would at least try to understand if it's possible. | 11:28 |
| panda | derekh: we already did some conversion by ourselves, so it would't fall completely on you | 11:29 |
| openstackgerrit | Michele Baldessari proposed openstack/puppet-pacemaker master: Initial pcs 0.10 support https://review.openstack.org/607873 | 11:29 |
| openstackgerrit | Michele Baldessari proposed openstack/puppet-pacemaker master: Initial support for properties managed by pcs 0.10 https://review.openstack.org/626005 | 11:29 |
| derekh | panda: great, what do you need to know | 11:30 |
| openstackgerrit | Marios Andreou proposed openstack-infra/tripleo-ci master: Add designate for scenario3 standalone tempest config https://review.openstack.org/633919 | 11:30 |
| openstackgerrit | Quique Llorente proposed openstack/tripleo-quickstart master: Use force_tcg by default https://review.openstack.org/633444 | 11:34 |
| *** avivgt has quit IRC | 11:34 | |
| *** salmankhan has quit IRC | 11:35 | |
| openstackgerrit | Michele Baldessari proposed openstack/puppet-tripleo master: Set pacemaker cluster_members_addr https://review.openstack.org/633578 | 11:36 |
| openstackgerrit | Derek Higgins proposed openstack/tripleo-quickstart-extras master: Setup Ironic in Overcloud https://review.openstack.org/509728 | 11:38 |
| panda | derekh: scenario012 installs and tests ironic on the overcloud, standalone uses a single node which doesn't map 1:1 to an undercloud. How much of this is applicable to standalone ? would and undercloud install be better at testing ironic instead ? | 11:39 |
| openstackgerrit | Quique Llorente proposed openstack/tripleo-quickstart-extras master: Add overcloud-scripts tag to network env task https://review.openstack.org/602025 | 11:39 |
| derekh | panda: well we were specifically trying to test that ironic in the overcloud works, would testing it in the undercloud essentially be the same as a OVB job? | 11:41 |
| openstackgerrit | Cédric Jeanneret proposed openstack/tripleo-quickstart-extras master: Add output for ip6tables and related ipv6 things https://review.openstack.org/633922 | 11:48 |
| Tengu | jaosorior: ramishra -^^ | 11:48 |
| Tengu | still... we don't see any "timeout" with ansible or anything else. really weird. | 11:49 |
| panda | derekh: yes, but the idea of standalone is that containers are essentially the same for undercloud and overcloud, so we test only once for both situations. | 11:52 |
| panda | derekh: do you think this applies to ironic ? | 11:52 |
| *** moguimar has joined #tripleo | 11:58 | |
| *** dprince has joined #tripleo | 11:59 | |
| *** dsneddon has joined #tripleo | 12:00 | |
| derekh | panda: perhaps, its not just the containers that we are testing though, we'd also need to ensure the same heat templates, networks and some other stuff (probably) are all exercised in the same way they would have been on the overcloud, so maybe we need a checklist or something | 12:00 |
| *** sdake has joined #tripleo | 12:02 | |
| *** jpena is now known as jpena|lunch | 12:04 | |
| panda | derekh: ok, it doesn't seem ready to be converted at this point, so I'll add a task on this sprint to aid in the preparation and this checklist. Are you ok with being the point of contact for this ? | 12:05 |
| derekh | panda: yup, sounds good | 12:06 |
| panda | derekh: thanks. | 12:06 |
| derekh | panda: thanks also ;-) | 12:06 |
| openstackgerrit | Tom Barron proposed openstack/tripleo-common master: Bump the Ceph container image to 3.2.1 https://review.openstack.org/630925 | 12:06 |
| *** shyamb has joined #tripleo | 12:07 | |
| *** dsneddon has quit IRC | 12:07 | |
| *** gkadam has joined #tripleo | 12:08 | |
| openstackgerrit | Dan Prince proposed openstack/python-tripleoclient master: Drop unused utils.py roles functions https://review.openstack.org/633810 | 12:08 |
| openstackgerrit | Raviv proposed openstack/tripleo-upgrade stable/queens: Add cli_opts_params.yaml to update prepare/converge. https://review.openstack.org/633911 | 12:11 |
| *** ansmith has joined #tripleo | 12:17 | |
| *** gkadam is now known as gkadam-bmgr | 12:19 | |
| shyamb | Hi | 12:20 |
| shyamb | We wanted to qualify triliovault with RHOSP14 | 12:20 |
| shyamb | just wanted to know, anything changed from deployment perspective? | 12:21 |
| *** janki has quit IRC | 12:21 | |
| openstackgerrit | Merged openstack/ansible-role-tripleo-modify-image master: Track obsoleted packages replacements for updates https://review.openstack.org/633484 | 12:23 |
| *** shyamb has quit IRC | 12:24 | |
| *** shyamb has joined #tripleo | 12:25 | |
| *** shyamb has quit IRC | 12:30 | |
| *** avivgt has joined #tripleo | 12:36 | |
| *** dsneddon has joined #tripleo | 12:39 | |
| openstackgerrit | Chandan Kumar proposed openstack-infra/tripleo-ci master: Run tempest using os_tempest role in standalone job https://review.openstack.org/627500 | 12:41 |
| *** sdake has quit IRC | 12:43 | |
| *** quiquell is now known as quiquell|lunch | 12:45 | |
| *** sdake has joined #tripleo | 12:45 | |
| *** dsneddon has quit IRC | 12:46 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-common master: Make python uploader the default https://review.openstack.org/616019 | 12:46 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-common master: Make python uploader the default https://review.openstack.org/616019 | 12:46 |
| ramishra | jaosorior, Tengu: https://bugs.launchpad.net/tripleo/+bug/1813900/comments/6 | 12:47 |
| openstack | Launchpad bug 1813900 in tripleo "tripleo-ci-centos-7-undercloud-containers timing out in host configuration for step 2" [Critical,In progress] - Assigned to Cédric Jeanneret (cjeanner) | 12:47 |
| *** dsneddon has joined #tripleo | 12:48 | |
| *** trown|outtypewww is now known as trown | 12:48 | |
| Tengu | ramishra: hmm. not sure if it's an issue as well -.-'. | 12:49 |
| *** avivgt has quit IRC | 12:51 | |
| Tengu | ramishra: plus, it's not really related to the *host* configuration steps... | 12:53 |
| *** morazi has joined #tripleo | 12:54 | |
| *** rlandy has joined #tripleo | 12:54 | |
| *** holser|lunch is now known as holser_ | 12:54 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Create tripleo-admin user on the undercloud https://review.openstack.org/633850 | 12:54 |
| *** dsneddon has quit IRC | 12:55 | |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-docs master: Move CI docs from contributor guide into new CI guide https://review.openstack.org/633771 | 12:56 |
| *** quiquell|lunch is now known as quiquell | 12:57 | |
| ramishra | Tengu: May be not. But ansible does not proceed beyond that host config step. That's something different in the logs from a successful run, some configuration may be causing that which is also resulting in deployment not proceeding further? | 13:00 |
| ramishra | bbiab | 13:01 |
| Tengu | ramishra: so I might suspect ip6table issue - so that's why I pushed the patch in order to get its state in the logs. | 13:02 |
| *** jpena|lunch is now known as jpena | 13:02 | |
| *** paramite has quit IRC | 13:04 | |
| *** dsneddon has joined #tripleo | 13:05 | |
| *** avivgt has joined #tripleo | 13:06 | |
| *** waleedm has joined #tripleo | 13:06 | |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart-extras master: Design new role for zuul-based reproducer https://review.openstack.org/631067 | 13:07 |
| *** rh-jelabarre has joined #tripleo | 13:07 | |
| ska | Is the purpose of the 2nd Nic in Overcloud hosts to given them access to the internet, or is it to give administrators an external access to those hosts ? | 13:08 |
| *** ooolpbot has joined #tripleo | 13:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1813900 | 13:10 |
| openstack | Launchpad bug 1813900 in tripleo "tripleo-ci-centos-7-undercloud-containers timing out in host configuration for step 2" [Critical,In progress] - Assigned to Cédric Jeanneret (cjeanner) | 13:10 |
| *** ooolpbot has quit IRC | 13:10 | |
| *** dsneddon has quit IRC | 13:14 | |
| mschuppert | hi, could I please get reviews on https://review.openstack.org/#/c/603343/ | 13:15 |
| *** ykarel is now known as ykarel|away | 13:17 | |
| openstackgerrit | Cédric Jeanneret proposed openstack/tripleo-quickstart-extras master: Add output for ip6tables and related ipv6 things https://review.openstack.org/633922 | 13:21 |
| Tengu | EmilienM: -^ looping. but nor really convinced. | 13:23 |
| EmilienM | Tengu: looking | 13:23 |
| Tengu | nope, looping | 13:23 |
| * Tengu runs away | 13:23 | |
| *** waleedm has quit IRC | 13:24 | |
| EmilienM | Tengu: yeah, you can go back to PS1, I'll +2 | 13:26 |
| EmilienM | Tengu: was just trying to avoid code dup | 13:26 |
| Tengu | EmilienM: :) | 13:29 |
| openstackgerrit | Cédric Jeanneret proposed openstack/tripleo-quickstart-extras master: Add output for ip6tables and related ipv6 things https://review.openstack.org/633922 | 13:29 |
| ykarel|away | Tengu, ramishra is the undercloud containers issue sorted? | 13:30 |
| ykarel|away | may be it's related to puppet-certmonger update 21 hours ago | 13:30 |
| Tengu | ykarel|away: ah, any hint on that one? | 13:30 |
| Tengu | for now we didn't find the real root cause. | 13:30 |
| ykarel|away | Tengu, https://review.rdoproject.org/r/#/c/18605 | 13:30 |
| ykarel|away | jaosorior, ^^ | 13:30 |
| jaosorior | ykarel|away: you think so? | 13:31 |
| ykarel|away | jaosorior, didn't looked much, but seen passing vs failing job | 13:32 |
| ykarel|away | and it looked suspicious | 13:32 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: WIP: remove postsave command from haproxy https://review.openstack.org/633940 | 13:33 |
| jaosorior | ykarel|away: lets give it a try ^^ | 13:33 |
| openstackgerrit | Adriano Petrich proposed openstack/tripleo-quickstart-extras master: Fix md5 for cirrus 3.6 image https://review.openstack.org/633941 | 13:34 |
| *** mcornea has joined #tripleo | 13:36 | |
| *** _fragatina has quit IRC | 13:38 | |
| *** _fragatina has joined #tripleo | 13:38 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo master: Only request certs in step 1 https://review.openstack.org/633944 | 13:39 |
| jaosorior | ykarel|away: this might also help ^^ if that's the case | 13:40 |
| *** dsneddon has joined #tripleo | 13:41 | |
| ykarel|away | jaosorior, ack | 13:42 |
| * ykarel|away leaving | 13:42 | |
| *** psachin has quit IRC | 13:44 | |
| bogdando | jaosorior, Tengu: the diff looks not very suspicious https://github.com/saltedsignal/puppet-certmonger/compare/ef9457ca650b652547fa99644b8a27de01b3db50..a19887066c4978ad2567c881d60c14fcf0737bb2 | 13:46 |
| Tengu | hmmm. puppet-stdlib version changes. | 13:46 |
| Tengu | oh. not really. hm. | 13:47 |
| *** pcaruana has quit IRC | 13:50 | |
| *** dsneddon has quit IRC | 13:50 | |
| *** dsneddon has joined #tripleo | 13:51 | |
| *** psachin has joined #tripleo | 13:53 | |
| *** ykarel|away has quit IRC | 13:56 | |
| *** dsneddon has quit IRC | 13:57 | |
| *** pcaruana has joined #tripleo | 13:57 | |
| *** psachin has quit IRC | 13:58 | |
| *** vinaykns has joined #tripleo | 13:59 | |
| *** psachin has joined #tripleo | 14:00 | |
| *** ekultails has joined #tripleo | 14:00 | |
| *** jcoufal has joined #tripleo | 14:02 | |
| *** janki has joined #tripleo | 14:03 | |
| bogdando | jaosorior, Tengu: hm, so https://review.openstack.org/#/c/633944/1/manifests/profile/base/certmonger_user.pp fixes some of the unwanted haproxy disruption for each step? | 14:09 |
| bogdando | may be it also fixes https://bugs.launchpad.net/tripleo/+bug/1813900 | 14:09 |
| openstack | Launchpad bug 1813900 in tripleo "tripleo-ci-centos-7-undercloud-containers timing out in host configuration for step 2" [Critical,In progress] - Assigned to Cédric Jeanneret (cjeanner) | 14:09 |
| *** ooolpbot has joined #tripleo | 14:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1813900 | 14:10 |
| *** ooolpbot has quit IRC | 14:10 | |
| jaosorior | bogdando: that's the hope | 14:10 |
| bogdando | good catch | 14:10 |
| *** tzumainn has joined #tripleo | 14:12 | |
| *** sdake has quit IRC | 14:20 | |
| gpocentek | Hi all | 14:25 |
| *** dsneddon has joined #tripleo | 14:26 | |
| gpocentek | do you have a policy for backports? I was considering backporting this change: https://review.openstack.org/#/c/623459/ | 14:26 |
| jaosorior | gpocentek: it's fine as long as it's a bug fix. | 14:28 |
| *** ykarel|away has joined #tripleo | 14:29 | |
| trozet_ | hjensas: ping? | 14:33 |
| *** dsneddon has quit IRC | 14:33 | |
| *** sdake has joined #tripleo | 14:34 | |
| gpocentek | jaosorior: should I just push a review on the stable/rocky branch then? | 14:35 |
| jaosorior | gpocentek: sure | 14:36 |
| gpocentek | great, thanks | 14:36 |
| *** ykarel|away is now known as ykarel | 14:36 | |
| openstackgerrit | David Peacock proposed openstack/tripleo-heat-templates master: flatten database service MySQL Client https://review.openstack.org/626760 | 14:39 |
| openstackgerrit | Gauvain Pocentek proposed openstack/tripleo-common stable/rocky: Install ipmitool in the mistral_executor image https://review.openstack.org/633965 | 14:40 |
| *** skramaja has quit IRC | 14:42 | |
| *** mjturek has joined #tripleo | 14:44 | |
| *** Vorrtex has joined #tripleo | 14:45 | |
| gpocentek | another topic: I'm working on a multi-region setup and my process for now is to create the users and endpoints in Keystone in an existing deployment, and then play with the EndpointMap to get the second region services to talk to Keystone | 14:45 |
| gpocentek | have you heard of a better to handle multi-region? | 14:46 |
| jaosorior | gpocentek: zzzeek was working on that ^^ ; but his work was postponed to the next release. | 14:47 |
| gpocentek | ok | 14:48 |
| *** quiquell is now known as quiquell|off | 14:48 | |
| *** mjturek has quit IRC | 14:48 | |
| openstackgerrit | mathieu bultel proposed openstack/tripleo-common stable/rocky: Publish error for package_update workflow https://review.openstack.org/633966 | 14:48 |
| *** sdake has quit IRC | 14:49 | |
| gpocentek | I'd be happy to help, with my very limited TripleO knowledge | 14:50 |
| *** aufi has joined #tripleo | 14:50 | |
| gpocentek | jaosorior: thanks for your answers! :) | 14:54 |
| jaosorior | np | 14:55 |
| *** sdake has joined #tripleo | 14:55 | |
| hjensas | trozet_: Hi | 14:56 |
| trozet_ | hjensas: hey man, hit a new networking related issue on master | 14:56 |
| trozet_ | hjensas: Error validating environment for plan overcloud: ERROR: Property error: resources.CephStorageIpListMap.properties: Unknown Property StorageMgmtIpList | 14:57 |
| trozet_ | hjensas: i tried to remove the network from roles, didn't seem to matter | 14:57 |
| trozet_ | hjensas: no matter what I tried, it seemed to always require storagemgmt resource mapped. Were there any related changes in the last week or so? | 14:59 |
| openstackgerrit | mathieu bultel proposed openstack/tripleo-common stable/queens: Publish error for package_update workflow https://review.openstack.org/633970 | 14:59 |
| openstackgerrit | mathieu bultel proposed openstack/tripleo-common stable/queens: Publish error for package_update workflow https://review.openstack.org/633970 | 15:00 |
| openstackgerrit | mathieu bultel proposed openstack/tripleo-common stable/queens: Publish error for package_update workflow https://review.openstack.org/633970 | 15:02 |
| hjensas | trozet_: https://review.openstack.org/627420 | 15:05 |
| hjensas | trozet_: 8 days ago ... | 15:05 |
| *** dsneddon has joined #tripleo | 15:05 | |
| hjensas | trozet_: The issue is here: https://github.com/openstack/tripleo-heat-templates/blob/20ef4456565f37c6f9035876f2fea588aea1c057/overcloud.j2.yaml#L549-L555 | 15:05 |
| trozet_ | hjensas: in my network datat he network is disabled | 15:06 |
| hjensas | The jinja there fall's back to passing the property even if the network is disabled. | 15:06 |
| openstackgerrit | Rabi Mishra proposed openstack/tripleo-quickstart-extras master: Fix md5 checksum for cirros-0.3.6 https://review.openstack.org/633972 | 15:07 |
| trozet_ | hjensas: so that's a bug right? | 15:07 |
| ykarel | ramishra, duplicated of https://review.openstack.org/#/c/633941/ | 15:07 |
| trozet_ | hjensas: so I created a new roles file and only had Controller and Compute in it, i got a similar error | 15:08 |
| trozet_ | hjensas: Unless i mapped the storagemgmt network to the right template, then it got passed it (even though the network was not enabled on the role or in network data) | 15:08 |
| *** udesale has joined #tripleo | 15:08 | |
| *** ooolpbot has joined #tripleo | 15:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1813900 | 15:10 |
| openstack | Launchpad bug 1813900 in tripleo "tripleo-ci-centos-7-undercloud-containers timing out in host configuration for step 2" [Critical,In progress] - Assigned to Cédric Jeanneret (cjeanner) | 15:10 |
| *** ooolpbot has quit IRC | 15:10 | |
| ramishra | ykarel: Ah, I was late. Probably filing a bug would have helped. I'll abandon, thanks! | 15:10 |
| hjensas | trozet_: yes, I think we can remove what is in the else there now that we don't template disable networks anymore. | 15:10 |
| hjensas | trozet_: but it's a bug. indeed. | 15:11 |
| openstackgerrit | Michele Baldessari proposed openstack/puppet-tripleo master: WIP DNR Unify rabbitmq bind mounts https://review.openstack.org/605599 | 15:11 |
| *** sdake has quit IRC | 15:11 | |
| trozet_ | hjensas: you want me to push a patch then pull it into OPNFV CI and we can see if it works? | 15:11 |
| *** dsneddon has quit IRC | 15:11 | |
| ykarel | ramishra, /me saw some disucssion some time back on #oooq, and when i saw ur patch, checked open reviews | 15:12 |
| hjensas | trozet_: sure, I was about to open a bug and then push a patch. But go ahead. | 15:12 |
| trozet_ | hjensas: ok | 15:12 |
| *** sdake has joined #tripleo | 15:14 | |
| ramishra | ykarel: thanks!, I commented on that to add the bug atleast:) | 15:15 |
| *** dciabrin has quit IRC | 15:15 | |
| ykarel | ramishra, good | 15:15 |
| *** dciabrin has joined #tripleo | 15:16 | |
| hjensas | trozet_: Filed a bug: https://bugs.launchpad.net/tripleo/+bug/1813950 | 15:16 |
| openstack | Launchpad bug 1813950 in tripleo " Error validating environment for plan overcloud: ERROR: Property error: resources.CephStorageIpListMap.properties: Unknown Property StorageMgmtIpList" [Undecided,Triaged] | 15:16 |
| trozet_ | hjensas: oh | 15:16 |
| trozet_ | hjensas: ok then you can just push the patch too if you want | 15:17 |
| hjensas | trozet_: ok. | 15:17 |
| trozet_ | hjensas: add me to it so i can pull it into my ci | 15:17 |
| openstackgerrit | Michele Baldessari proposed openstack/tripleo-heat-templates master: WIP DNR Unify rabbitmq bind mounts and env vars https://review.openstack.org/605598 | 15:17 |
| *** mdbooth has joined #tripleo | 15:18 | |
| mdbooth | ykarel: Just looking at https://review.openstack.org/#/c/620561/ | 15:18 |
| mdbooth | Which I believe you used to test https://review.openstack.org/#/c/626952/ | 15:19 |
| *** quiquell|off has quit IRC | 15:19 | |
| ykarel | mdbooth, yes ^^ + some other things | 15:19 |
| mdbooth | ykarel: I'm about to make a significant change to that patch. Any chance you could update your change to have 2 Depends-On lines so I can just run recheck on it? | 15:20 |
| mdbooth | Looks like you're explicitly patching in run_tests.sh | 15:20 |
| ykarel | mdbooth, yes depends-on on projects do not work, https://review.openstack.org/#/c/620561/17/run_tests.sh@221 | 15:21 |
| ykarel | need to update run_tests | 15:21 |
| mdbooth | ykarel: Ah, ok. That's annoying. Can I ping you when it's done? | 15:21 |
| ykarel | mdbooth, sure | 15:21 |
| mdbooth | ykarel: Thanks. | 15:22 |
| mdbooth | Just waiting on a local run of functional to complete, then I'll push it. | 15:22 |
| openstackgerrit | Harald Jensås proposed openstack/tripleo-heat-templates master: Fix network IpListMap - no longer need fallback https://review.openstack.org/633975 | 15:23 |
| ykarel | mdbooth, ack | 15:24 |
| hjensas | trozet_: ^^ Try that, hopefully it will fix the issue. | 15:24 |
| trozet_ | hjensas: ok, this will pull it in and test it: https://gerrit.opnfv.org/gerrit/#/c/66789/1 | 15:29 |
| *** quiquell has joined #tripleo | 15:33 | |
| *** quiquell is now known as quiquell|off | 15:33 | |
| *** wolverineav has joined #tripleo | 15:36 | |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart master: Use force_tcg by default https://review.openstack.org/633444 | 15:39 |
| openstackgerrit | David Peacock proposed openstack/tripleo-heat-templates master: WIP - flatten database service MySQL Server https://review.openstack.org/633980 | 15:39 |
| mdbooth | ykarel: I've just pushed it. I should probably wait until it's passed nova CI before asking you to update your test. | 15:40 |
| *** wolverineav has quit IRC | 15:41 | |
| *** dsneddon has joined #tripleo | 15:41 | |
| ykarel | mdbooth, i am pushing update to run_tests.sh | 15:46 |
| ykarel | i will be away in some time as nova ci may take some time | 15:46 |
| ykarel | you can push patch if required to run_tests.sh | 15:46 |
| *** janki has quit IRC | 15:47 | |
| *** dsneddon has quit IRC | 15:47 | |
| ykarel | mdbooth, pushed https://review.openstack.org/620561 | 15:47 |
| ykarel | mdbooth, you can update https://review.openstack.org/#/c/620561/18/run_tests.sh@221 | 15:47 |
| ykarel | ^^ line if required | 15:48 |
| mdbooth | ykarel: I actually have to update that patch again :/ | 15:57 |
| mdbooth | ykarel: Thanks, will do. | 15:57 |
| dciabrin | EmilienM, hey how do I update a service's password on a containerized undercloud? do I change undercloud-passwords.conf and "openstack undercloud upgrade"? or do I pass it via a new .yaml file? | 15:57 |
| *** gkadam-bmgr has quit IRC | 15:57 | |
| EmilienM | dciabrin: updated undercloud-passwords.conf should work, if not we probably have a bug | 15:57 |
| EmilienM | updating* | 15:58 |
| *** arxcruz|ruck is now known as arxcruz | 15:58 | |
| dciabrin | EmilienM, ok I'll try right away thx | 15:58 |
| EmilienM | dciabrin: lmk how it goes | 15:59 |
| *** pcaruana has quit IRC | 16:01 | |
| ykarel | jaosorior, so undercloud-containers job moved to validate-tempest on your patch removing certmonger postcmd from haproxy:- https://zuul.openstack.org/stream/e18b0d5fa06b41fc9e94aa092964f3d5?logfile=console.log | 16:01 |
| ykarel | Tengu, ^^ | 16:01 |
| jaosorior | ykarel: OK... so that's an issue it seems. | 16:01 |
| jaosorior | ykarel: what about the other patch? | 16:01 |
| jaosorior | that moves the certmonger requests to only happen in step 1 ? | 16:01 |
| *** saneax has quit IRC | 16:01 | |
| ykarel | jaosorior, didn't checked that | 16:01 |
| *** _fragatina has quit IRC | 16:01 | |
| *** udesale has quit IRC | 16:02 | |
| jaosorior | ykarel: for me that would be preferred (if that works.. hope it does) | 16:02 |
| ykarel | jaosorior, still running | 16:02 |
| ykarel | it's at undercloud deploy: https://zuul.openstack.org/stream/304bf55cfedd4c2592c3c8f907977247?logfile=console.log | 16:02 |
| ykarel | atleast cause is clear | 16:03 |
| ykarel | jaosorior, so is this change causing the issue? | 16:04 |
| ykarel | - request_args << "\"#{resource[:postsave_cmd]}\"" | 16:04 |
| ykarel | + request_args << "#{resource[:postsave_cmd]}" | 16:04 |
| ykarel | ^^ one in puppet-certmonger | 16:04 |
| jaosorior | ykarel: yeah, that seems to be the issue | 16:04 |
| jaosorior | ykarel: it was an issue that postsave commands were not running at all | 16:04 |
| jaosorior | which is problematic | 16:04 |
| jaosorior | now that they're running, it seems they run too often | 16:04 |
| ykarel | okk | 16:05 |
| zzzeek | jaosorior gpocentek that work isn't going to happen unless I can get people to finally approve it, I spent so much time on phone calls, going to denver to get signoff and then still, the same people wouldnt sign off on the spec | 16:05 |
| jaosorior | zzzeek: they were asking for more details on the usecase though (on the spec) | 16:06 |
| zzzeek | jaosorior: I gave all I had. fabio had set me on that work and he left the group | 16:06 |
| jaosorior | zzzeek: understood. Sorry about the outcome | 16:07 |
| zzzeek | jaosorior: the work is there and we can still integrate it, i just was trying to be doing it in a vacuum. as always if folks want this, I can pick it right up | 16:08 |
| zzzeek | err to "not be doing it in a vacuum" | 16:08 |
| zzzeek | jaosorior: im sort of on the side that thinks is a little nuts as well, is the problem :) keystones just federating woudl be a lot easier to maintain | 16:09 |
| zzzeek | no way do i want a customer call to fix that setup without a lot of other people who are invested in how it works | 16:10 |
| *** ooolpbot has joined #tripleo | 16:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1813900 | 16:10 |
| *** ooolpbot has quit IRC | 16:10 | |
| openstack | Launchpad bug 1813900 in tripleo "tripleo-ci-centos-7-undercloud-containers timing out in host configuration for step 2" [Critical,In progress] - Assigned to Cédric Jeanneret (cjeanner) | 16:10 |
| *** agopi is now known as agopi|food | 16:11 | |
| *** dsneddon has joined #tripleo | 16:14 | |
| *** sdake has quit IRC | 16:14 | |
| *** ccamacho has quit IRC | 16:16 | |
| *** pcaruana has joined #tripleo | 16:17 | |
| *** kopecmartin is now known as kopecmartin|off | 16:20 | |
| *** ssbarnea|rover has quit IRC | 16:20 | |
| *** ssbarnea has joined #tripleo | 16:21 | |
| *** sdake has joined #tripleo | 16:21 | |
| *** yprokule has quit IRC | 16:25 | |
| openstackgerrit | Nagasai Vinaykumar Kapalavai proposed openstack/puppet-tripleo master: Qdr's in public interface https://review.openstack.org/634004 | 16:25 |
| gpocentek | zzzeek: out of curiosity is the goal to have 1 undercloud to rule all the overclouds? Or is it multiple undercloud "working together"? | 16:26 |
| openstackgerrit | Nagasai Vinaykumar Kapalavai proposed openstack/tripleo-heat-templates master: Qdr's in a public interface https://review.openstack.org/634005 | 16:27 |
| zzzeek | gpocentek: there's talk that one undercloud should contorl multiple overclouds but i dont know when they want to get that going | 16:27 |
| zzzeek | gpocentek: feel free to comment on my spec at https://review.openstack.org/#/c/600555/ if you have notes on rationale | 16:27 |
| * zzzeek is actually on PTO. going to the pool :) | 16:27 | |
| gpocentek | zzzeek: I'll have a look at the spec and see how I can provide some feedback | 16:29 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-docs master: Add notes for DCN operational availability modes https://review.openstack.org/632089 | 16:30 |
| *** dtantsur is now known as dtantsur|afk | 16:32 | |
| EmilienM | mandre: hey, do you have a min to talk about kolla & buildah? | 16:38 |
| *** ramishra has quit IRC | 16:40 | |
| openstackgerrit | Kamil Sambor proposed openstack/tripleo-quickstart master: Skip test not supported by OVN https://review.openstack.org/633524 | 16:42 |
| openstackgerrit | Kamil Sambor proposed openstack/tripleo-heat-templates master: Switch default neutron ML2 mechanism driver to OVN https://review.openstack.org/593056 | 16:43 |
| *** ksambor has quit IRC | 16:44 | |
| *** pcaruana has quit IRC | 16:45 | |
| *** psachin has quit IRC | 16:48 | |
| *** aufi has quit IRC | 16:49 | |
| *** suuuper has quit IRC | 16:49 | |
| *** sdake has quit IRC | 16:51 | |
| *** thrash is now known as thrash|g0ne | 16:53 | |
| *** jfrancoa has quit IRC | 16:54 | |
| openstackgerrit | Cédric Jeanneret proposed openstack/tripleo-quickstart-extras master: Add output for ip6tables and related ipv6 things https://review.openstack.org/633922 | 16:57 |
| *** sdake has joined #tripleo | 17:00 | |
| openstackgerrit | Adriano Petrich proposed openstack/tripleo-quickstart-extras master: Fix md5 for cirrus 3.6 image https://review.openstack.org/633941 | 17:01 |
| *** radeks_ has quit IRC | 17:02 | |
| *** radeks_ has joined #tripleo | 17:02 | |
| *** florianf has quit IRC | 17:03 | |
| *** ykarel is now known as ykarel|away | 17:03 | |
| *** moguimar has quit IRC | 17:04 | |
| openstackgerrit | Pradeep Kilambi proposed openstack/puppet-tripleo master: Add a connector to Qpid router https://review.openstack.org/611979 | 17:05 |
| *** _fragatina has joined #tripleo | 17:06 | |
| *** ooolpbot has joined #tripleo | 17:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1813900 | 17:10 |
| openstack | Launchpad bug 1813900 in tripleo "tripleo-ci-centos-7-undercloud-containers timing out in host configuration for step 2" [Critical,In progress] - Assigned to Cédric Jeanneret (cjeanner) | 17:10 |
| *** ooolpbot has quit IRC | 17:10 | |
| *** bnemec has quit IRC | 17:14 | |
| mandre | EmilienM: hey! what's up? | 17:15 |
| EmilienM | mandre: with mwhahaha we were looking at how we can get rid of docker for image building | 17:15 |
| EmilienM | mandre: we think there are at least 2 options | 17:16 |
| EmilienM | mandre: 1) Add buildah support to kolla build code (sounds painful, buildah has no python library now) | 17:16 |
| EmilienM | mandre: or use template-only option and deal the build workflow on our side, but we have to deal with the ordering, and such | 17:16 |
| EmilienM | wdyt? | 17:17 |
| mandre | EmilienM: will buildah take Dockerfile as input? | 17:17 |
| EmilienM | yes | 17:17 |
| mandre | perfect, then you can tell kolla to only generate the Dockerfiles and then build the images using buildah, would that work for you? | 17:17 |
| *** noama has quit IRC | 17:18 | |
| EmilienM | mandre: yes I think we'll take this path | 17:18 |
| *** agopi|food is now known as agopi | 17:18 | |
| mandre | EmilienM: seems like the most straighforward | 17:18 |
| *** jpich has quit IRC | 17:20 | |
| mandre | EmilienM: you need to pass the --template-only opt to the build command, and a --work-dir too otherwise it will put the generated Dockerfiles in a tempdir IIRC | 17:24 |
| EmilienM | ok | 17:24 |
| EmilienM | mandre: we were wondering about the order | 17:24 |
| EmilienM | to build base first, etc | 17:24 |
| mandre | EmilienM: you can also generate this with kolla-build | 17:25 |
| EmilienM | mandre: without docker? | 17:27 |
| mandre | EmilienM: normally yes, but just tested it and it tried to connect to the docker socket -- it should be fairly easy to fix | 17:28 |
| EmilienM | mandre: what the command would o? | 17:29 |
| EmilienM | would do* | 17:29 |
| *** wolverineav has joined #tripleo | 17:30 | |
| *** derekh has quit IRC | 17:33 | |
| *** wolverineav has quit IRC | 17:34 | |
| mandre | EmilienM: ok, I fixed the bug and now I can build the dependency tree without docker running | 17:35 |
| mandre | EmilienM: use the --list-dependencies opt | 17:35 |
| EmilienM | mandre: I need to play with that | 17:36 |
| trozet_ | hjensas: now undercloud install is failing in master: https://paste.fedoraproject.org/paste/uNIzsdphLiFP4RaVFYE9lA/raw | 17:37 |
| *** cylopez has quit IRC | 17:38 | |
| mandre | EmilienM: actually, --list-images will also sort images in a way that sorts with parents first so you could potentially also use it if easier for you | 17:39 |
| bogdando | mandre, EmilienM, mwhahaha: right, just that that "ordering" of dockerfiles is not a very simple task | 17:40 |
| bogdando | I have some WIP scripts | 17:40 |
| * mwhahaha is aware | 17:40 | |
| bogdando | but that's very hacky bash | 17:40 |
| * mwhahaha doesn't wish to reimplement this stuff | 17:40 | |
| mandre | bogdando: kolla already solved this problem | 17:40 |
| * mwhahaha is busy hating life elsewhere | 17:40 | |
| bogdando | mandre: not for the case we have it only generated dockerfiles | 17:40 |
| bogdando | then we're on our own how to assemble that pile of layers | 17:41 |
| bogdando | but if anyone wants to try, I have some notes | 17:41 |
| mandre | bogdando: see above discussion with EmilienM, kolla can generate a dependency tree for the images and also list images in an order where parents are before children | 17:41 |
| bogdando | http://rhel-ha.etherpad.corp.redhat.com/rhel8-openstack-ha-container-build (sorry that's an internal ehterpad link) | 17:41 |
| bogdando | mandre: ack | 17:42 |
| bogdando | something to start with indeed | 17:42 |
| hjensas | trozet_: hm, I guess tripleo-common does'nt check if the network is enabled or not. I'll take a look. | 17:42 |
| trozet_ | hjensas: https://bugs.launchpad.net/tripleo/+bug/1813972 | 17:42 |
| openstack | Launchpad bug 1813972 in tripleo "Undercloud install fails in write_static_inventory with Exception list index out of range " [Undecided,New] | 17:42 |
| mandre | bogdando: I just need to push a simple patch to make it work without docker running | 17:42 |
| bogdando | mandre: nice, thanks! | 17:45 |
| *** bogdando has quit IRC | 17:49 | |
| mandre | EmilienM: fyi https://review.openstack.org/#/c/634017/ lets you run the commands I mentioned without docker running | 17:51 |
| mandre | or installed | 17:51 |
| EmilienM | mandre: pretty cool, thanks | 17:51 |
| EmilienM | mandre: I think the build.py will need a huge refacto, when we have a replacement for docker (buildah probably) | 17:52 |
| EmilienM | but in the meantime I think we'll change our image build process | 17:52 |
| *** dmacpher has quit IRC | 17:52 | |
| mandre | build.py needs a huge refactor no matter what :) | 17:52 |
| *** dmacpher has joined #tripleo | 17:56 | |
| *** nkinder has quit IRC | 17:59 | |
| openstackgerrit | Jill Rouleau proposed openstack/tripleo-heat-templates master: step1: flatten nova service configuration https://review.openstack.org/632522 | 18:03 |
| *** xek_ has joined #tripleo | 18:08 | |
| *** dsneddon has quit IRC | 18:08 | |
| *** ooolpbot has joined #tripleo | 18:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1813900 | 18:10 |
| openstack | Launchpad bug 1813900 in tripleo "tripleo-ci-centos-7-undercloud-containers timing out in host configuration for step 2" [Critical,In progress] - Assigned to Cédric Jeanneret (cjeanner) | 18:10 |
| *** ooolpbot has quit IRC | 18:10 | |
| *** saneax has joined #tripleo | 18:10 | |
| *** trown is now known as trown|lunch | 18:10 | |
| *** xek has quit IRC | 18:10 | |
| *** hakhande_ has quit IRC | 18:12 | |
| *** jpena is now known as jpena|off | 18:13 | |
| *** wolverineav has joined #tripleo | 18:13 | |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart-extras master: Design new role for zuul-based reproducer https://review.openstack.org/631067 | 18:14 |
| *** wolverineav has quit IRC | 18:15 | |
| *** shardy has quit IRC | 18:15 | |
| *** wolverin_ has joined #tripleo | 18:15 | |
| hjensas | trozet_: It looks like stack output of RoleNetHostnameMap is still have the disabled networks. Or it has networks not associated with the role in roles data? | 18:15 |
| *** wolverin_ has quit IRC | 18:15 | |
| *** wolverineav has joined #tripleo | 18:16 | |
| *** dciabrin has quit IRC | 18:16 | |
| *** dciabrin has joined #tripleo | 18:16 | |
| *** pcaruana has joined #tripleo | 18:17 | |
| hjensas | trozet_: afaict the templates are filtering on enabled networks and role networks for this value - see: puppet/role.role.j2.yaml:820 | 18:19 |
| hjensas | trozet_: Unfortunately I have to run now, but I can pick this up tomorrow. | 18:20 |
| *** eggs has quit IRC | 18:22 | |
| *** saneax has quit IRC | 18:23 | |
| openstackgerrit | Jill Rouleau proposed openstack/tripleo-heat-templates master: step1: flatten nova service configuration https://review.openstack.org/632522 | 18:24 |
| openstackgerrit | Jill Rouleau proposed openstack/tripleo-heat-templates master: step2: flatten nova service configuration https://review.openstack.org/633278 | 18:24 |
| *** eggs has joined #tripleo | 18:24 | |
| *** sshnaidm is now known as sshnaidm|afk | 18:25 | |
| *** wolverineav has quit IRC | 18:27 | |
| *** morazi has quit IRC | 18:29 | |
| *** mmethot has quit IRC | 18:30 | |
| *** mmethot has joined #tripleo | 18:30 | |
| *** morazi has joined #tripleo | 18:31 | |
| *** wolverineav has joined #tripleo | 18:31 | |
| *** ekultails has quit IRC | 18:33 | |
| *** dsneddon has joined #tripleo | 18:34 | |
| jaosorior | EmilienM: mind if we merge this one https://review.openstack.org/#/c/633944/ ? it addresses the timeouts https://bugs.launchpad.net/tripleo/+bug/1813900 | 18:42 |
| openstack | Launchpad bug 1813900 in tripleo "tripleo-ci-centos-7-undercloud-containers timing out in host configuration for step 2" [Critical,In progress] - Assigned to Cédric Jeanneret (cjeanner) | 18:42 |
| *** dsneddon has quit IRC | 18:42 | |
| EmilienM | jaosorior: interesting | 18:42 |
| EmilienM | are we sure? | 18:42 |
| EmilienM | jaosorior: I don't see it in commit msg | 18:43 |
| jaosorior | Seems it might be related to the request being issued at every step | 18:43 |
| jaosorior | oh | 18:43 |
| jaosorior | crap | 18:43 |
| openstackgerrit | David Peacock proposed openstack/tripleo-heat-templates master: flatten database service MySQL Server https://review.openstack.org/633980 | 18:49 |
| *** wolverineav has quit IRC | 18:53 | |
| *** wolverineav has joined #tripleo | 18:53 | |
| *** mmethot has quit IRC | 18:55 | |
| *** wolverineav has quit IRC | 18:56 | |
| *** wolverineav has joined #tripleo | 18:56 | |
| *** mdbooth_ has joined #tripleo | 18:57 | |
| *** moguimar has joined #tripleo | 18:59 | |
| *** mdbooth has quit IRC | 18:59 | |
| *** moguimar has quit IRC | 19:00 | |
| *** dsneddon has joined #tripleo | 19:03 | |
| *** ykarel|away has quit IRC | 19:05 | |
| *** morazi has quit IRC | 19:08 | |
| *** ooolpbot has joined #tripleo | 19:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1813900 | 19:10 |
| *** ooolpbot has quit IRC | 19:10 | |
| openstack | Launchpad bug 1813900 in tripleo "tripleo-ci-centos-7-undercloud-containers timing out in host configuration for step 2" [Critical,In progress] - Assigned to Cédric Jeanneret (cjeanner) | 19:10 |
| *** avivgt has quit IRC | 19:10 | |
| *** ekultails has joined #tripleo | 19:11 | |
| *** paramite has joined #tripleo | 19:12 | |
| *** trown|lunch is now known as trown | 19:23 | |
| *** wolverineav has quit IRC | 19:24 | |
| *** wolverineav has joined #tripleo | 19:32 | |
| *** mmethot has joined #tripleo | 19:33 | |
| *** wolverineav has quit IRC | 19:37 | |
| jschlueter | mwhahaha: hmm placement-api container build is failing for https://review.openstack.org/#/c/633906/5 | 19:40 |
| jschlueter | dangling symlink for healthcheck | 19:41 |
| * mwhahaha shrugs | 19:41 | |
| * jschlueter remembers something that way coming across recently ... | 19:41 | |
| jschlueter | will dig on it | 19:41 |
| mwhahaha | dangling health check usually means a mismatchin tripleo-common | 19:42 |
| mwhahaha | so the one being used in the containers with is older/newer than the one that provides the override file | 19:42 |
| jschlueter | https://review.openstack.org/#/c/632057/ | 19:42 |
| mwhahaha | if you're using my patch, it might need a rebasing | 19:42 |
| * jschlueter sighs | 19:42 | |
| jschlueter | it's kolla patch ... so likely mismatch between promoted and what's landed | 19:43 |
| jschlueter | lyarwood: ^^ | 19:43 |
| mwhahaha | yea that'll break things too | 19:43 |
| * mwhahaha blames EmilienM for ignoring the tripleo-build-containers-centos-7 job | 19:43 | |
| jschlueter | non-voting job in kolla is failing due to mismatch between what is merged and what is in promoted hash | 19:44 |
| mwhahaha | kolla is failing cause of percona stuff | 19:44 |
| * jschlueter sighs | 19:44 | |
| mwhahaha | teh health check items is all tripleo | 19:44 |
| EmilienM | oops | 19:44 |
| jschlueter | it is but that has a patch and I was looking at what other jobs are failing including non-voting and saw this | 19:44 |
| jschlueter | mwhahaha: review 633906 is the fix for percona in kolla | 19:45 |
| mwhahaha | right | 19:45 |
| mwhahaha | but that only affects centos | 19:45 |
| mwhahaha | not fedora builds | 19:45 |
| mwhahaha | probably affects RHEL for silly reasons | 19:45 |
| jschlueter | I'm following because it's blocking my other kolla cleanup patches as well as your python3 one | 19:46 |
| jschlueter | lyarwood, EmilienM, mwhahaha: second patch in a week that has broken non-voting job in kolla ... we should look at how we can prevent the noise and still get these things landed cleaner | 19:49 |
| *** wolverineav has joined #tripleo | 19:49 | |
| mwhahaha | well the fix is to make that job voting | 19:50 |
| mwhahaha | but we have to fix it first | 19:50 |
| mwhahaha | also have more time in the day | 19:50 |
| jschlueter | http://logs.openstack.org/57/632057/1/check/tripleo-build-containers-centos-7/2dba738/logs/build-err.log.txt.gz | 19:51 |
| jschlueter | this job did fail on that patch | 19:51 |
| mwhahaha | yea i'm aware | 19:51 |
| mwhahaha | but it's non-voting so it didn't block anything | 19:51 |
| * jschlueter sighs | 19:51 | |
| mwhahaha | hence the problem with non-voting jobs | 19:51 |
| * jschlueter sighs | 19:52 | |
| *** panda is now known as panda|off | 19:59 | |
| *** fmount has quit IRC | 20:03 | |
| *** fmount has joined #tripleo | 20:04 | |
| openstackgerrit | David Peacock proposed openstack/tripleo-heat-templates master: flatten database service MySQL Server https://review.openstack.org/633980 | 20:05 |
| *** sdake has quit IRC | 20:06 | |
| *** sdake has joined #tripleo | 20:06 | |
| openstackgerrit | Emilien Macchi proposed openstack/ansible-role-container-registry master: docker: add reset_connection https://review.openstack.org/634041 | 20:09 |
| *** ooolpbot has joined #tripleo | 20:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1813900 | 20:10 |
| openstack | Launchpad bug 1813900 in tripleo "tripleo-ci-centos-7-undercloud-containers timing out in host configuration for step 2" [Critical,In progress] - Assigned to Cédric Jeanneret (cjeanner) | 20:10 |
| *** ooolpbot has quit IRC | 20:10 | |
| openstackgerrit | Emilien Macchi proposed openstack/ansible-role-container-registry master: docker: add reset_connection https://review.openstack.org/634041 | 20:10 |
| *** rfolco has quit IRC | 20:13 | |
| weshay | mwhahaha, +1 on making container-build voting if you want to | 20:20 |
| jidar | fultonj: my deploy directory is missing the osd scenario :( https://gist.github.com/67c8482647c0708575f59fb2bddf9d9e | 20:21 |
| weshay | any have any chrony failures recently? | 20:22 |
| fultonj | jidar: https://docs.openstack.org/tripleo-docs/latest/install/advanced_deployment/ceph_config.html#override-ansible-run-options | 20:24 |
| *** rfolco has joined #tripleo | 20:24 | |
| fultonj | jidar: please read the paragraph that starts with "During the playbook run temporary files, like the Ansible inventory and the ceph-ansible parameters that are passed as overrides " | 20:24 |
| fultonj | and look in the described /tmp to see what ceph-ansible is actually getting | 20:25 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart master: write out the libvirt node ip addresses https://review.openstack.org/633843 | 20:25 |
| fultonj | so you can rule out any heat passing issues | 20:25 |
| *** dprince has quit IRC | 20:25 | |
| fultonj | or heat override issues | 20:26 |
| jidar | its damn near empty | 20:26 |
| fultonj | jidar: there are multiples | 20:27 |
| fultonj | jidar: one is for the ansible playbook to create the UUID | 20:27 |
| fultonj | the ohter is for the ceph-ansible run | 20:27 |
| fultonj | you want the one with the latest ceph-ansible run | 20:27 |
| fultonj | have a look at the ceph-ansible inventory as it should contain the variables | 20:27 |
| jidar | hmm, ok | 20:28 |
| jidar | fultonj: https://gist.github.com/5dbc2b300ace3db598eeb546f3287dfc | 20:33 |
| fultonj | ansible-mistral-actionGc9sZ_ is for the get uuid playbook | 20:34 |
| fultonj | you want the other one | 20:34 |
| fultonj | find inventory.yaml | 20:34 |
| * fultonj has to join a phone call | 20:34 | |
| jidar | okie dokie | 20:34 |
| jidar | all of my inventory.yaml files look the same | 20:35 |
| fultonj | 'get uuid' --> get the system uuid via dmidecode | 20:35 |
| jidar | also I noticed something, the storage-environment.yaml is the only one that really references the osd_scenero, but I'm not including it in my deploy command.. is that a problem? | 20:36 |
| * fultonj on call | 20:36 | |
| *** sdake has quit IRC | 20:39 | |
| *** rfolco has quit IRC | 20:40 | |
| *** sdake has joined #tripleo | 20:42 | |
| *** sdake has quit IRC | 20:44 | |
| *** xek_ has quit IRC | 20:45 | |
| *** xek_ has joined #tripleo | 20:46 | |
| *** sdake has joined #tripleo | 20:46 | |
| *** gfidente has quit IRC | 20:48 | |
| *** paramite has quit IRC | 20:50 | |
| *** KingJ has quit IRC | 20:54 | |
| *** KingJ has joined #tripleo | 20:55 | |
| *** ekultails has quit IRC | 20:57 | |
| *** wolverineav has quit IRC | 20:59 | |
| *** hberaud is now known as hberaud|gone | 21:01 | |
| *** sdake has quit IRC | 21:01 | |
| *** sdake has joined #tripleo | 21:02 | |
| *** xek_ has quit IRC | 21:03 | |
| *** fmount has quit IRC | 21:04 | |
| *** fmount has joined #tripleo | 21:05 | |
| *** ekultails has joined #tripleo | 21:05 | |
| *** Vorrtex has quit IRC | 21:07 | |
| *** wolverineav has joined #tripleo | 21:07 | |
| *** ooolpbot has joined #tripleo | 21:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1813900 | 21:10 |
| openstack | Launchpad bug 1813900 in tripleo "tripleo-ci-centos-7-undercloud-containers timing out in host configuration for step 2" [Critical,In progress] - Assigned to Cédric Jeanneret (cjeanner) | 21:10 |
| *** ooolpbot has quit IRC | 21:10 | |
| openstackgerrit | Jill Rouleau proposed openstack/tripleo-heat-templates master: step2: flatten nova service configuration https://review.openstack.org/633278 | 21:15 |
| *** radeks_ has quit IRC | 21:19 | |
| *** fmount has quit IRC | 21:28 | |
| openstackgerrit | Jill Rouleau proposed openstack/tripleo-heat-templates master: step3: flatten nova service configuration https://review.openstack.org/633597 | 21:28 |
| *** fmount has joined #tripleo | 21:29 | |
| openstackgerrit | Jill Rouleau proposed openstack/tripleo-heat-templates master: step1: flatten nova service configuration https://review.openstack.org/632522 | 21:31 |
| *** nkinder has joined #tripleo | 21:31 | |
| *** holser_ has quit IRC | 21:36 | |
| *** sdake has quit IRC | 21:38 | |
| *** cylopez has joined #tripleo | 21:42 | |
| *** nkinder has quit IRC | 21:44 | |
| openstackgerrit | Steve Baker proposed openstack/tripleo-common master: Break out tripleo-admin creation to its own role https://review.openstack.org/633827 | 21:45 |
| *** sdake has joined #tripleo | 21:49 | |
| *** openstackgerrit has quit IRC | 21:50 | |
| *** openstackgerrit has joined #tripleo | 21:57 | |
| openstackgerrit | Steve Baker proposed openstack/tripleo-heat-templates master: Create tripleo-admin user on the undercloud https://review.openstack.org/633850 | 21:57 |
| *** ianw_pto is now known as ianw | 21:59 | |
| *** ekultails has quit IRC | 22:02 | |
| *** trown is now known as trown|outtypewww | 22:07 | |
| *** ooolpbot has joined #tripleo | 22:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1813900 | 22:10 |
| openstack | Launchpad bug 1813900 in tripleo "tripleo-ci-centos-7-undercloud-containers timing out in host configuration for step 2" [Critical,In progress] - Assigned to Cédric Jeanneret (cjeanner) | 22:10 |
| *** ooolpbot has quit IRC | 22:10 | |
| ska | I get this error when tyring to build/upload image: (undercloud) [stack@kvm-undercloud:~]: openstack overcloud image upload | 22:12 |
| ska | Required file "./ironic-python-agent.initramfs" does not exist. | 22:12 |
| jidar | whats the exact command you're using | 22:13 |
| ska | openstack overcloud image build;openstack overcloud image upload ; (from inside stack user's env) | 22:16 |
| jidar | and where are your images? | 22:16 |
| jidar | I've always had to pass it the param | 22:17 |
| ska | I don't see them. I see some stuff in cache but looks incomplete. | 22:18 |
| ska | Following https://docs.openstack.org/tripleo-docs/latest/install/basic_deployment/basic_deployment_cli.html | 22:19 |
| jidar | from that page: > Alternatively, images are available via RDO at https://images.rdoproject.org/master which offers images from both the CentOS Build System (cbs) and RDO Trunk (called rdo_trunk or delorean). However this mirror is slow so if you experience slow download speeds you should skip to building the images instead. | 22:19 |
| openstackgerrit | Jill Rouleau proposed openstack/tripleo-heat-templates master: step1: flatten nova service configuration https://review.openstack.org/632522 | 22:19 |
| ska | jidar: should i just download and unpack an image bundle/ | 22:20 |
| ska | ? | 22:20 |
| jidar | its up to you, I've done both, building can be a bit of a pain | 22:21 |
| ska | I passed the --no-skip option to the builder and its cranking.. we'll see. | 22:22 |
| ska | Its installing a ton of packages, I hope I didn't fat-finger it. | 22:23 |
| ska | What do I grab on the archive site, a tgz? | 22:25 |
| jidar | dunno, it's been a while | 22:25 |
| *** dtrainor has quit IRC | 22:26 | |
| ska | I wonder why they don't ocmpress those tar images.. Would save a lot of IO. | 22:26 |
| ska | Where do I look for the images if they build correctly? in ~stack/* ? | 22:30 |
| ska | I'm seeing stuff now in ~stack/* so I guess its happier with me. | 22:31 |
| ska | I wish I know what i did to piss if off the first time around. | 22:32 |
| openstackgerrit | Steve Baker proposed openstack/tripleo-heat-templates master: Create tripleo-admin user on the undercloud https://review.openstack.org/633850 | 22:39 |
| mwhahaha | i don't think qcow2 images compress well | 22:43 |
| *** cfontain has joined #tripleo | 22:50 | |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-common master: Remove placement healthcheck from overrides https://review.openstack.org/634059 | 22:54 |
| openstackgerrit | Jill Rouleau proposed openstack/tripleo-heat-templates master: flatten ceilometer service configuration https://review.openstack.org/633617 | 22:58 |
| openstackgerrit | Alex Schultz proposed openstack-infra/tripleo-ci master: Make container build job voting https://review.openstack.org/634060 | 22:58 |
| *** mcornea has quit IRC | 22:58 | |
| openstackgerrit | Alex Schultz proposed openstack-infra/tripleo-ci master: Make container build job voting https://review.openstack.org/634060 | 22:58 |
| mwhahaha | weshay, EmilienM -^ | 22:59 |
| EmilienM | yes | 22:59 |
| EmilienM | let's do that | 22:59 |
| mwhahaha | so why are we hitting a bunch of TIMED_OUTs over the last two days | 23:00 |
| EmilienM | jaosorior said he had a patch but I didn't get it | 23:01 |
| EmilienM | for certmonger step | 23:01 |
| mwhahaha | https://review.openstack.org/#/c/633944/ | 23:01 |
| mwhahaha | that one? | 23:01 |
| *** tkajinam has joined #tripleo | 23:01 | |
| *** sdake has quit IRC | 23:01 | |
| EmilienM | yes | 23:01 |
| mwhahaha | did we get a new certmonger that's breaking things or something? | 23:01 |
| mwhahaha | or i wonder if it's related to all thsoe restart patches we merged (?) | 23:02 |
| EmilienM | mwhahaha: look at your 11:42:14 here | 23:02 |
| EmilienM | juan pinged me on this one | 23:02 |
| * EmilienM reads bug again | 23:02 | |
| EmilienM | mwhahaha: maybe https://review.openstack.org/#/c/623353/ ? | 23:03 |
| mwhahaha | https://review.openstack.org/#/c/623353/ | 23:03 |
| mwhahaha | yea that thing | 23:03 |
| EmilienM | so yeah maybe the step patch will help? | 23:04 |
| EmilienM | who knows | 23:04 |
| mwhahaha | we'll see | 23:04 |
| * EmilienM invokes the stars | 23:04 | |
| mwhahaha | i guess | 23:04 |
| EmilienM | guessing is a safe bet in a retry/sleep driven deployment | 23:04 |
| * mwhahaha dies a bit more inside | 23:05 | |
| mwhahaha | speaking of retires | 23:05 |
| mwhahaha | let's go work on buildah | 23:05 |
| *** vinaykns has quit IRC | 23:05 | |
| EmilienM | yeah so | 23:05 |
| EmilienM | we need to hack in the tripleoclient | 23:05 |
| *** dtrainor has joined #tripleo | 23:06 | |
| * EmilienM opens https://github.com/openstack/python-tripleoclient/blob/master/tripleoclient/v1/container_image.py for the first time | 23:06 | |
| *** toure is now known as toure|gone | 23:07 | |
| *** rascasoft has quit IRC | 23:07 | |
| mwhahaha | it's actually in tripleo-common | 23:07 |
| mwhahaha | i think | 23:07 |
| mwhahaha | it's all int he kolla_builder | 23:07 |
| EmilienM | https://github.com/openstack/python-tripleoclient/blob/master/tripleoclient/v1/container_image.py#L175-L203 | 23:07 |
| EmilienM | I think we'll need to hack in there | 23:08 |
| EmilienM | that's where KollaImageBuilder is called at least | 23:08 |
| EmilienM | it's actually BuildImage.images_from_deps iiuc | 23:09 |
| *** dtrainor has quit IRC | 23:09 | |
| *** ooolpbot has joined #tripleo | 23:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1813900 | 23:10 |
| openstack | Launchpad bug 1813900 in tripleo "tripleo-ci-centos-7-undercloud-containers timing out in host configuration for step 2" [Critical,In progress] - Assigned to Cédric Jeanneret (cjeanner) | 23:10 |
| *** ooolpbot has quit IRC | 23:10 | |
| mwhahaha | of course juan's patch timed out in tempest | 23:10 |
| * EmilienM timeouts itself | 23:11 | |
| EmilienM | yeah you're right | 23:12 |
| EmilienM | it's in tripleo-common | 23:12 |
| *** dtrainor has joined #tripleo | 23:12 | |
| EmilienM | mwhahaha: https://github.com/openstack/tripleo-common/blob/master/tripleo_common/image/kolla_builder.py#L435-L462 | 23:13 |
| EmilienM | mwhahaha: i'm wondering if it wouldn't be easier to just use your ansible role | 23:14 |
| EmilienM | instead of python | 23:14 |
| EmilienM | which would make this process easier to reproduce outside of tripleoclient | 23:14 |
| EmilienM | and we could still keep the tripleoclient command but it would call ansible | 23:14 |
| mwhahaha | no my ansible role uses this code | 23:14 |
| *** wolverineav has quit IRC | 23:15 | |
| EmilienM | mwhahaha: right but | 23:15 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-common master: Revert "Remove placement healthcheck from overrides" https://review.openstack.org/634064 | 23:15 |
| EmilienM | mwhahaha: your ansible role could call kolla-build | 23:15 |
| mwhahaha | it's more complicated than that | 23:16 |
| mwhahaha | because we have python to tell us what subset of container we need from kolla | 23:16 |
| mwhahaha | trust me, fixing up the python is better | 23:16 |
| EmilienM | fine | 23:16 |
| mwhahaha | and by better i mean, i'd rather put hot sauce in my eyeballs than do this | 23:17 |
| * mwhahaha cries | 23:17 | |
| *** wolverineav has joined #tripleo | 23:20 | |
| *** dtrainor has quit IRC | 23:20 | |
| *** cfontain has quit IRC | 23:21 | |
| *** rlandy is now known as rlandy|bbl | 23:24 | |
| openstackgerrit | Chris Krelle proposed openstack/tripleo-docs master: correct vlan example when setting up overcloud network https://review.openstack.org/634065 | 23:24 |
| *** wolverineav has quit IRC | 23:25 | |
| *** dtrainor has joined #tripleo | 23:25 | |
| NobodyCam | morning hey hey folks just checking: we should be using neutron net-create here? https://docs.openstack.org/tripleo-docs/latest/install/advanced_deployment/network_isolation.html#using-the-native-vlan-on-a-trunked-interface | 23:28 |
| *** sdake has joined #tripleo | 23:29 | |
| *** wolverineav has joined #tripleo | 23:31 | |
| openstackgerrit | Alex Schultz proposed openstack-infra/tripleo-ci master: Fix repo configuration https://review.openstack.org/633527 | 23:32 |
| jidar | gonna go grab dinner and work on this a little more, gfidente fultonj if you guys are up to helping out a little bit more | 23:33 |
| *** sdake has quit IRC | 23:33 | |
| openstackgerrit | Alex Schultz proposed openstack-infra/tripleo-ci master: Make container build job voting https://review.openstack.org/634060 | 23:37 |
| *** slaweq has quit IRC | 23:38 | |
| *** cfontain has joined #tripleo | 23:50 | |
| openstackgerrit | Alex Schultz proposed openstack-infra/tripleo-ci master: Fix repo configuration https://review.openstack.org/633527 | 23:52 |
| *** rascasoft has joined #tripleo | 23:55 | |
| openstackgerrit | Alex Schultz proposed openstack-infra/tripleo-ci master: Make container build job voting https://review.openstack.org/634060 | 23:58 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!