Wednesday, 2019-03-20

« Tuesday, 2019-03-19 Index Thursday, 2019-03-21 »
*** lemko has quit IRC00:31
*** jamesgu has quit IRC00:55
*** cheng1 has joined #airshipit01:26
*** jamesgu has joined #airshipit03:16
*** sthussey has quit IRC03:35
*** cheng1 has quit IRC04:41
*** cheng1 has joined #airshipit05:23
*** kranthikirang has joined #airshipit05:34
*** kranthikirang has quit IRC05:38
*** jamesgu has quit IRC05:47
*** kranthikirang has joined #airshipit07:22
*** kranthikirang has quit IRC07:26
*** roman_g has joined #airshipit07:40
*** lemko has joined #airshipit08:34
*** skatsaounis has quit IRC08:42
*** kranthikirang has joined #airshipit09:10
*** kranthikirang has quit IRC09:14
*** debugger_io has joined #airshipit09:46
*** juhak has quit IRC09:46
*** juhak has joined #airshipit09:47
*** cheng1 has quit IRC09:49
*** cheng1 has joined #airshipit10:06
*** debugger_io has quit IRC10:09
openstackgerritSmruti Soumitra Khuntia proposed openstack/airship-armada master: End user logging for audit traceabilty  https://review.openstack.org/63812410:47
*** kranthikirang has joined #airshipit10:58
openstackgerritSmruti Soumitra Khuntia proposed openstack/airship-shipyard master: User context tracing through logging  https://review.openstack.org/63387311:00
openstackgerritSmruti Soumitra Khuntia proposed openstack/airship-in-a-bottle master: Document End user optional header  https://review.openstack.org/64299911:01
openstackgerritSmruti Soumitra Khuntia proposed openstack/airship-in-a-bottle master: Document End user optional header  https://review.openstack.org/64299911:02
*** kranthikirang has quit IRC11:03
openstackgerritHemanth Nakkina proposed openstack/airship-in-a-bottle master: Add seccomp profile on genesis node in multinode gate  https://review.openstack.org/64482411:07
openstackgerritSmruti Soumitra Khuntia proposed openstack/airship-armada master: End user logging for audit traceabilty  https://review.openstack.org/63812411:12
openstackgerritSmruti Soumitra Khuntia proposed openstack/airship-armada master: End user logging for audit traceabilty  https://review.openstack.org/63812411:17
*** Zara has left #airshipit11:22
openstackgerritSmruti Soumitra Khuntia proposed openstack/airship-shipyard master: User context tracing through logging  https://review.openstack.org/63387311:27
openstackgerritSmruti Soumitra Khuntia proposed openstack/airship-armada master: End user logging for audit traceabilty  https://review.openstack.org/63812411:38
*** cheng1 has quit IRC11:39
*** Nishant_ has joined #airshipit11:47
*** juhak has quit IRC12:02
*** juhak has joined #airshipit12:02
openstackgerritSmruti Soumitra Khuntia proposed openstack/airship-armada master: End user logging for audit traceabilty  https://review.openstack.org/63812412:41
*** cheng1 has joined #airshipit12:42
*** kranthikirang has joined #airshipit12:46
*** cheng1 has quit IRC12:47
*** kranthikirang has quit IRC12:51
*** skatsaounis has joined #airshipit12:55
*** aaronsheffield has joined #airshipit13:26
*** sthussey has joined #airshipit13:34
openstackgerritScott Hussey proposed openstack/airship-in-a-bottle master: (multinode) Make disk layout flexible  https://review.openstack.org/63804013:44
*** kranthikirang has joined #airshipit13:45
*** dustinspecker has joined #airshipit13:50
*** michael-beaver has joined #airshipit14:30
*** jamesgu has joined #airshipit14:32
*** kranthikirang has quit IRC14:48
evgenylWe've had a problem with Artifactory license keys as a result logs for AIAB gates were not available for some time, now it should be back online.15:06
*** michaelbeaver has joined #airshipit15:15
*** michael-beaver has quit IRC15:18
*** altlogbot_0 has joined #airshipit15:28
*** altlogbot_0 has quit IRC15:30
*** dustinspecker has quit IRC15:36
roman_gPlease, review https://review.openstack.org/#/c/635507/ - openstack/airship-treasuremap - Add cache for results of requests to quay.io in Updater tool15:48
roman_gPending review for quite a few weeks, has been advertised for review on weekly meetings a few times already.15:49
*** altlogbot_0 has joined #airshipit16:39
*** altlogbot_0 has quit IRC16:41
openstackgerritDrew Walters proposed openstack/airship-treasuremap master: airskiff: Reduce postgresql PVC size  https://review.openstack.org/63912316:42
openstackgerritDrew Walters proposed openstack/airship-treasuremap master: global: Add nfs-provisioner chart  https://review.openstack.org/63521716:42
openstackgerritDrew Walters proposed openstack/airship-treasuremap master: airskiff: Pull rather than build Airship images  https://review.openstack.org/63523116:42
openstackgerritDrew Walters proposed openstack/airship-treasuremap master: airskiff: Use global Airship charts  https://review.openstack.org/63521816:42
openstackgerritPRATEEK REDDY DODDA proposed openstack/airship-armada master: Implement Security Context for Armada  https://review.openstack.org/63920716:55
openstackgerritPRATEEK REDDY DODDA proposed openstack/airship-deckhand master: Implement Security Context for Deckhand  https://review.openstack.org/63920417:07
openstackgerritDrew Walters proposed openstack/airship-treasuremap master: airskiff: Use global Airship charts  https://review.openstack.org/63521817:20
*** mbeierl has quit IRC17:20
*** kranthikirang has joined #airshipit17:23
*** altlogbot_3 has joined #airshipit17:23
*** altlogbot_3 has quit IRC17:24
*** altlogbot_3 has joined #airshipit17:25
*** altlogbot_3 has quit IRC17:28
*** altlogbot_3 has joined #airshipit17:29
openstackgerritPRATEEK REDDY DODDA proposed openstack/airship-armada master: Implement Security Context for Armada  https://review.openstack.org/63920717:29
openstackgerritMerged openstack/airship-divingbell master: Run Divingbell containers as unprivileged  https://review.openstack.org/63943517:31
openstackgerritMerged openstack/airship-in-a-bottle master: Minor fixes as follow-up for new Pegleg  https://review.openstack.org/64466417:37
*** mbeierl has joined #airshipit17:53
*** mbeierl has quit IRC17:58
openstackgerritLev Morgan proposed openstack/airship-pegleg master: Additional fix to secrets linting  https://review.openstack.org/64495818:05
*** mbeierl has joined #airshipit18:14
*** irclogbot_0 has quit IRC18:21
*** irclogbot_2 has joined #airshipit18:23
openstackgerritLev Morgan proposed openstack/airship-pegleg master: Fix typo in secrets linting  https://review.openstack.org/64495818:27
*** irclogbot_2 has quit IRC18:27
*** irclogbot_2 has joined #airshipit18:28
*** neith has quit IRC18:30
openstackgerritKaspars Skels proposed openstack/airship-treasuremap master: global: Add nfs-provisioner chart  https://review.openstack.org/63521718:30
*** aaronsheffield has quit IRC18:30
*** evgenyl has quit IRC18:30
*** b-str has quit IRC18:30
*** aaronsheffield has joined #airshipit18:31
*** neith has joined #airshipit18:31
*** v1k0d3n has quit IRC18:31
*** pas-ha has quit IRC18:31
*** evgenyl has joined #airshipit18:32
*** pas-ha has joined #airshipit18:34
*** v1k0d3n has joined #airshipit18:34
openstackgerritDrew Walters proposed openstack/airship-treasuremap master: airskiff: Use Minikube for Kubernetes deployment  https://review.openstack.org/62261018:36
openstackgerritDrew Walters proposed openstack/airship-treasuremap master: airskiff: Use global Airship charts  https://review.openstack.org/63521818:42
openstackgerritDrew Walters proposed openstack/airship-treasuremap master: airskiff: Pull rather than build Airship images  https://review.openstack.org/63523118:43
evgenylHi everyone, need some help with promenade/k8s bootstrap process, we have failures on our baremetal treasuremap lab, during genesis deployment k8s API is not available through HAproxy, the script that configures HAproxy fails http://paste.openstack.org/show/748135/ because it cannot connect to k8s API, and this endpoint is unavailable, because k8s-proxy cannot access HAproxy http://paste.openstack.org/show/748136/ , we get18:55
evgenylit reproduced for a few days already, what else should I look into to figure what is wrong?18:55
evgenylAnd is the following endpoint handled via kube-proxy? `/etc/genesis/armada/assets/manifest.yaml:1064:        kubernetes_url: https://10.96.0.1:443`18:57
*** altlogbot_3 has quit IRC19:00
*** altlogbot_3 has joined #airshipit19:01
sthusseyWhat changed?19:01
evgenylComparing with the latest tag there was haproxy uplift and tls cipher whitelisting https://github.com/openstack/airship-treasuremap/compare/v19.03.06..master19:05
*** altlogbot_3 has quit IRC19:06
*** altlogbot_3 has joined #airshipit19:06
sthusseyI'm not sure. Do you redeploy genesis OS?19:12
*** irclogbot_2 has quit IRC19:16
*** irclogbot_3 has joined #airshipit19:16
openstackgerritSandeep Reddy Thumma proposed openstack/airship-promenade master: Set Least Previliage for .PEM files under directory /etc/genesis.  https://review.openstack.org/64077519:17
evgenylsthussey: No redeployment from scratch, there are a bunch of scripts that do cleanup19:19
evgenylsthussey: I see there are two haproxy pods, haproxy and haproxy-anchor, as far as I can see "haproxy" just watches for changes in the config file and restarts the service if needed and "haproxy-anchor" gets up to date endpoints and updates the config, is this correct?19:19
sthusseythe anchor actually puts the haproxy static pod in place19:20
sthusseyas well as managing the config file19:20
sthusseyI think the thought is that the haproxy static pod put in place during genesis should be good enough to bootstrap kube-proxy19:20
sthusseyand then going forward the haproxy anchor can use the kube-proxy managed service IP to access the kubernetes API19:20
sthusseybut honestly I had no input to the anchor pattern19:21
sthusseyor the overall orchestration of cluster bootstrapping19:22
evgenylsthussey: Ok, now I have a better understanding of the flow, will have a closer look into that, thanks!19:23
*** rihbb has joined #airshipit19:24
rihbbHi, what is the difference between update_site & update_software scripts under shipyard/tools? For instance if I change kubernetes version in versions.yaml, do I have to run update_site or update_software to incorporate the changes? Thanks in advance.19:26
sthusseyupdate_site encompasses deploying additional nodes and updating charts19:26
sthusseyupdate_software only updates charts - and so would deploy your changes to kubernetes versions19:26
*** michael-beaver has joined #airshipit19:26
sthusseyhowever updating kubernetes versions isn't as simply as just updating versions.yaml19:27
sthusseybecause currently airship has no facility for updating the kubelet binary across the cluster19:27
evgenylrihbb: Yes, as sthussey mentioned there is no automatic update of kubelet, however it should be relatively easy to do by copying kubelet binary and doing reboot of the nodes one-by-one.19:29
*** michaelbeaver has quit IRC19:30
*** irclogbot_3 has quit IRC19:30
*** irclogbot_2 has joined #airshipit19:30
evgenylrihbb: Running `update_software` should be enough if you don't have new Nodes to be installed. `update_site` includes `update_software` and it uses drydock + maas to ensure that all nodes are provisioned, it should not hurt to run `update_site`, it just takes a bit more time than running `update_software`.19:33
sthusseyyou should update kubelet after the chart update finishes19:33
sthusseyold kubelet + new apiserver is fine, new kubelet + old apiserver is not19:33
openstackgerritAhmad Mahmoudi proposed openstack/airship-maas master: [DE6874] - Fixed maas-rack reschedule issue  https://review.openstack.org/64217419:34
*** altlogbot_3 has quit IRC19:34
*** altlogbot_3 has joined #airshipit19:35
rihbbsthussey, evgenyl: Thanks for the clarification. But when I change versions in version.yaml, running update_software doesn seem to recreate the pods even though the update_software script shows ¨Successfully performed update_software¨. Is there some other file that also needs to be changed apart from versions.yaml under global?19:35
rihbbevgenyl: this is in reference to the openstack update discussion we had few days back.19:36
rihbbDeploying airship with another version of openstack (apart from the default ocata) works only after redeploying the whole cluster & not by running update_software.19:38
rihbbIs there some other config that needs to be passed to reflect the new changes?19:40
openstackgerritSandeep Reddy Thumma proposed openstack/airship-promenade master: Set Least Previliage for .PEM files under directory /etc/genesis.  https://review.openstack.org/64077519:44
evgenylrihbb: Interesting, changing the images should work just fine, have you re-collected and re-uploaded updated yamls?19:47
openstackgerritPRATEEK REDDY DODDA proposed openstack/airship-deckhand master: Implement Security Context for Deckhand  https://review.openstack.org/63920419:49
openstackgerritMatt McEuen proposed openstack/airship-promenade master: Fix generate_certs typo  https://review.openstack.org/64498519:53
rihbbevgenyl: Yes - I did sudo airship-pegleg/tools/pegleg.sh site  -r airship-treasuremap collect $NEW_SITE -s ${NEW_SITE}_collected ; sudo -E airship-promenade/tools/simple-deployment.sh ${NEW_SITE}_collected ${NEW_SITE}_bundle & copied the newly generated treasuremap.yaml to the genesis node.20:13
rihbbHave I missed something?20:13
evgenylrihbb: Have you run `sudo -E airship-shipyard/tools/deckhand_load_yaml.sh ${NEW_SITE} ${NEW_SITE}_collected` from here https://airship-treasuremap.readthedocs.io/en/latest/authoring_and_deployment.html ?20:18
evgenylrihbb: And by the way, you don't need to run `airship-promenade/tools/simple-deployment.sh` in this case, change -> pegleg collect -> copy -> deckhand load -> software update.20:19
openstackgerritLev Morgan proposed openstack/airship-pegleg master: Fix typo in secrets linting  https://review.openstack.org/64495820:35
kranthikirangevgenyl: Where do we configure IPMI. username for servers? I see password is being configured in secrets/passphrases/ipmi_admin_password.yaml20:39
evgenylkranthikirang: Let me see, just a sec.20:39
sthusseyIn the HostProfile for the server20:40
evgenylkranthikirang: Here is a link https://github.com/openstack/airship-treasuremap/blob/master/global/profiles/host/cp.yaml#L2320:41
evgenylkranthikirang: And here you can also see how the passwords gets added into data -> oob tree https://github.com/openstack/airship-treasuremap/blob/master/global/profiles/host/cp.yaml#L12-L1820:43
kranthikirangevgenyl: strange; I am not seeing anything like this20:44
*** lemko has quit IRC20:44
kranthikirangWhen I clone I see data: has hardware_profile: dell_r72020:44
kranthikirangand the hardware profile contains only network and device class information20:45
sthusseyHostProfile20:45
*** irclogbot_2 has quit IRC20:45
kranthikirangGuess, I have to use that reference20:46
*** irclogbot_2 has joined #airshipit20:46
kranthikirangand create my profile for my own gen hO20:46
kranthikirangHP20:46
openstackgerritAhmad Mahmoudi proposed openstack/airship-maas master: [FIX] - Fixed maas-rack reschedule issue  https://review.openstack.org/64217420:47
evgenylkranthikirang: It's defined in globals https://github.com/openstack/airship-treasuremap/blob/master/global/profiles/host/cp.yaml#L23 and you can override it in your HW specific profile, see other examples here  https://github.com/openstack/airship-treasuremap/blob/master/site/airship-seaworthy/profiles/host/cp_r720.yaml#L14-L2220:47
kranthikirangevgenyl: cool, that means I should replace oob20:49
kranthikirang- method: replaceand path: .oob20:49
evgenylkranthikirang: To debug that it gets correctly replaced, I recommend using pegleg render command, you will be able to see all the values substituted https://airship-pegleg.readthedocs.io/en/latest/cli/cli.html#render20:49
evgenylkranthikirang: You can do a replace or you can use `merge` and replace an account name.20:50
kranthikirangevgenyl: ok20:50
evgenylkranthikirang: Here are a bit more details on how all this works https://airshipit.readthedocs.io/projects/deckhand/en/latest/users/layering.html?highlight=merge#action-types20:51
*** altlogbot_3 has quit IRC20:53
*** altlogbot_3 has joined #airshipit20:54
evgenylI'm still struggling with haproxy/k8s apiservers, so I see haproxy running with a single backend and apiserver is also running, but the config for haproxy is a bit weird `server s10.23.22.11 10.23.22.11:6444 check port 6443` for some reasons backend port is 6444, but check port is 6443, apiserver serves is running on 6444, as far as I understand `6444` port is used for initial k8s bootstrap and it gets changed to `6443`20:57
evgenylwhen run as "normal" pods instead of "static", any ideas on how this could happen?20:57
openstackgerritDmitrii Kabanov proposed openstack/airship-divingbell master: Add possibility to add repository and GPG key  https://review.openstack.org/64449520:57
openstackgerritDmitrii Kabanov proposed openstack/airship-divingbell master: Add possibility to add repository and GPG key  https://review.openstack.org/64449521:02
rihbbevgenyl: Thanks for pointing that out. Will try the k8s upgrade tomorrow :)21:07
evgenylrihbb: Cool, let us know how it goes :)21:08
*** mbologna has quit IRC21:12
rihbbevgenyl: bdw I have also seen ¨failed¨ and ¨upstream_failed¨ during update_software. DO you have an idea of what could go wrong?21:12
*** michaelbeaver has joined #airshipit21:12
rihbbOr how this could be debugged? As all pods/jobs seem to be in running state.21:14
evgenylrihbb: Can you send a complete output? You can use `./run_shipyard.sh get actions` and `./run_shipyard.sh describe action/ID` to show a bit more details.21:14
*** kranthikirang has quit IRC21:15
evgenylThose scripts should be in your airship-shipyard/tools  directory.21:15
*** michael-beaver has quit IRC21:15
rihbbevgenyl: This is how it looks like; https://paste.ubuntu.com/p/hj9m9tDp7S/21:17
*** michaelbeaver has quit IRC21:17
*** JosW has joined #airshipit21:18
*** JosW has left #airshipit21:18
evgenylrihbb: A generic debugging advice is when you see that `armada_build` failed, check armada-api logs in ucp namespace, there are two instances running, so you will need to check both, usually there is an error that says which set of helm-charts it failed to deploy, after that you will have some ideas which pod-logs to look at.21:19
evgenylrihbb: Armada is a kind of an orchestrator on top of helm, it basically pushes new releases to helm in a specific order and runs tests on every stage to ensure that everything works as expected.21:21
openstackgerritEvgeniy L proposed openstack/airship-treasuremap master: [WIP][DNM] test patch  https://review.openstack.org/64501221:25
rihbbevgenyl: Thanks for the tip. The logs of armada show ´2019-03-20 21:08:01.749 8 DEBUG armada.api.middleware [-] Response body:{"type": "error", "message": "Failed to apply manifest: Exception deploying charts: ['etcd']", "retry": false} process_response ./armada/api/middleware.py:127´21:26
rihbbAlso running ./tools/run_shipyard.sh throws the error ´FileNotFoundError: [Errno 2] No such file or directory: '/etc/shipyard/api-paste.ini'¨.21:27
evgenylrihbb: So looks like there are problems with deploying etcd, you can check that state of etcd pods, also it would be helpful to see more logs, like what was before and after this line.21:28
evgenylrihbb: Hm, can you try running it from tools dir?21:28
rihbbevgenyl: Running from tools directory throwed the error: ´Error: Unable to invoke action due to: The request you have made requires authentication. (HTTP 401) (Request-ID: req-522edf8b-cd15-4245-ae7a-5bca5578aae0)¨21:30
evgenylrihbb: Have you sourced shipyard creds?21:33
evgenyl`export OS_USERNAME=shipyard` and `export OS_PASSWORD=PASS` where pass is the value from `secrets/passphrases/ucp_shipyard_keystone_password.yaml`21:34
rihbbevgenyl: I see. The output of describe action/ID doesn give information about the error but armada logs show 2019-03-20 21:08:01.749 8 INFO armada.api.middleware [-] POST http://armada-api.ucp.svc.cluster.local:8000/api/v1.0/apply?tiller_host=10.97.222.252&target_manifest=full-site&tiller_port=44134 - 500 Internal Server Error21:40
evgenylrihbb: Can you send like 200 lines of logs and copy them here http://paste.openstack.org/ ?21:42
evgenylrihbb: I will need to drop, from your previous message I see that there was a problem deploying `etcd` so I recommend to check if all etcd pods are up and running. You can send a link to logs here, I will check those when I'm back tomorrow.21:45
openstackgerritLev Morgan proposed openstack/airship-pegleg master: Added cleartext option to passphrase generation  https://review.openstack.org/64501721:47
rihbbevgenyl: Thanks a lot. I will redo it tomorrow and will update accordingly. Have a nice rest of the day!21:47
*** rihbb has left #airshipit21:53
*** kranthikirang has joined #airshipit22:04
*** kranthikirang has quit IRC22:09
openstackgerritStacey Fletcher proposed openstack/airship-pegleg master: [DNM] Added cleartext option to passphrase generation  https://review.openstack.org/64501722:18
openstackgerritPRATEEK REDDY DODDA proposed openstack/airship-deckhand master: Implement Security Context for Deckhand  https://review.openstack.org/63920422:20
openstackgerritDan Crank proposed openstack/airship-deckhand master: Log client-id in UCP API endpoints  https://review.openstack.org/63406822:41
*** aaronsheffield has quit IRC23:03
*** sthussey has quit IRC23:53
openstackgerritRoman Gorshunov proposed openstack/airship-divingbell master: Minor: Git URI schema and URL change  https://review.openstack.org/64504123:58
« Tuesday, 2019-03-19 Index Thursday, 2019-03-21 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!