Planet RDO

August 13, 2019

RDO Blog

Community Blog Round Up 13 August 2019

Making Host and OpenStack iSCSI devices play nice together by geguileo

OpenStack services assume that they are the sole owners of the iSCSI connections to the iSCSI portal-targets generated by the Cinder driver, and that is fine 98% of the time, but what happens when we also want to have other non-OpenStack iSCSI volumes from that same storage system present on boot? In OpenStack the OS-Brick […]

Read more at https://gorka.eguileor.com/host-iscsi-devices/

Service Assurance on small OpenShift Cluster by mrunge

This article is intended to give an overview on how to test the

Read more at http://www.matthias-runge.de/2019/07/09/Service-Assurance-on-ocp/

Notes on testing a tripleo-common mistral patch by JohnLikesOpenStack

I recently ran into bug 1834094 and wanted to test the proposed fix. These are my notes if I have to do this again.

Read more at http://blog.johnlikesopenstack.com/2019/07/notes-on-testing-tripleo-common-mistral.html

Developer workflow with TripleO by Emilien

In this post we’ll see how one can use TripleO for developing & testing changes into OpenStack Python-based projects (e.g. Keystone).

Read more at https://my1.fr/blog/developer-workflow-with-tripleo/

Avoid rebase hell: squashing without rebasing by OddBit

You’re working on a pull request. You’ve been working on a pull request for a while, and due to lack of sleep or inebriation you’ve been merging changes into your feature branch rather than rebasing. You now have a pull request that looks like this (I’ve marked merge commits with the text [merge]):

Read more at https://blog.oddbit.com/post/2019-06-17-avoid-rebase-hell-squashing-wi/

Git Etiquette: Commit messages and pull requests by OddBit

Always work on a branch (never commit on master) When working with an upstream codebase, always make your changes on a feature branch rather than your local master branch. This will make it easier to keep your local master branch current with respect to upstream, and can help avoid situations in which you accidentally overwrite your local changes or introduce unnecessary merge commits into your history.

Read more at https://blog.oddbit.com/post/2019-06-14-git-etiquette-commit-messages/

Running Keystone with Docker Compose by OddBit

In this article, we will look at what is necessary to run OpenStack’s Keystone service (and the requisite database server) in containers using Docker Compose.

Read more at https://blog.oddbit.com/post/2019-06-07-running-keystone-with-docker-c/

The Kubernetes in a box project by Carlos Camacho

Implementing cloud computing solutions that runs in hybrid environments might be the final solution when comes to finding the best benefits/cost ratio.

Read more at https://www.anstack.com/blog/2019/05/21/kubebox.html

Running Relax-and-Recover to save your OpenStack deployment by Carlos Camacho

ReaR is a pretty impressive disaster recovery solution for Linux. Relax-and-Recover, creates both a bootable rescue image and a backup of the associated files you choose.

Read more at https://www.anstack.com/blog/2019/05/20/relax-and-recover-backups.html

by Rain Leander at August 13, 2019 08:00 AM

July 23, 2019

Gorka Eguileor

Making Host and OpenStack iSCSI devices play nice together

OpenStack services assume that they are the sole owners of the iSCSI connections to the iSCSI portal-targets generated by the Cinder driver, and that is fine 98% of the time, but what happens when we also want to have other non-OpenStack iSCSI volumes from that same storage system present on boot? In OpenStack the OS-Brick […]

by geguileo at July 23, 2019 05:49 PM

July 09, 2019

Matthias Runge

Service Assurance on small OpenShift Cluster

This article is intended to give an overview on how to test the Service Assurance Framework on a small deployment of OpenShift.

I've started with a deployment of RHEL 7.x on a baremetal machine.

After using subscribtion-manager to subscribe and attach, I made sure to have the necessary repositories …

by mrunge at July 09, 2019 02:50 PM

July 03, 2019

John Likes OpenStack

Notes on testing a tripleo-common mistral patch

I recently ran into bug 1834094 and wanted to test the proposed fix. These are my notes if I have to do this again.

Get a patched container

Because the mistral-executor is running as a container on the undercloud I needed to build a new container and TripleO's Container Image Preparationhelped me do this without too much trouble.

As described the Container Image Preparation docs, I already download a local copy of the containers to my undercloud by running the following:


time sudo openstack tripleo container image prepare \
-e ~/train/containers.yaml \
--output-env-file ~/containers-env-file.yaml
where ~/train/containers.yaml has the following:

---
parameter_defaults:
NeutronMechanismDrivers: ovn
ContainerImagePrepare:
- push_destination: 192.168.24.1:8787
set:
ceph_image: daemon
ceph_namespace: docker.io/ceph
ceph_tag: v4.0.0-stable-4.0-nautilus-centos-7-x86_64
name_prefix: centos-binary
namespace: docker.io/tripleomaster
tag: current-tripleo

I now want to download the same set of containers to my undercloud but I want the mistral-executor container to have the proposed fix. If I vist the review and click download I can see the patch is at refs/changes/60/668560/3 and I can pass this information to TripleO's Container Image Preparationso that it builds me a container with that patch applied.

To do this I update my containers.yaml to exclude the mistral-executor container from the usual tags with the excludes list directive and then create a separate section with the includes directive specific to the mistral-executor container.

Within this new section I ask that the tripleo-modify-image ansible role pull that patch and apply it to that source image.


---
parameter_defaults:
NeutronMechanismDrivers: ovn
ContainerImagePrepare:
- push_destination: 192.168.24.1:8787
set:
ceph_image: daemon
ceph_namespace: docker.io/ceph
ceph_tag: v4.0.0-stable-4.0-nautilus-centos-7-x86_64
name_prefix: centos-binary
namespace: docker.io/tripleomaster
tag: current-tripleo
excludes: [mistral-executor]
- push_destination: 192.168.24.1:8787
set:
name_prefix: centos-binary
namespace: docker.io/tripleomaster
tag: current-tripleo
modify_role: tripleo-modify-image
modify_append_tag: "-devel-ps3"
modify_vars:
tasks_from: dev_install.yml
source_image: docker.io/tripleomaster/centos-binary-mistral-executor:current-tripleo
refspecs:
-
project: tripleo-common
refspec: refs/changes/60/668560/3
includes: [mistral-executor]

When I then run the `sudo openstack tripleo container image prepare` command I see that it took a few extra steps to create my new container image.


Writing manifest to image destination
Storing signatures
INFO[0005] created - from /var/lib/containers/storage/overlay/10c5e9ec709991e7eb6cbbf99c08d87f9f728c1644d64e3b070bc3c81adcbc03/diff
and /var/lib/containers/storage/overlay-layers/10c5e9ec709991e7eb6cbbf99c08d87f9f728c1644d64e3b070bc3c81adcbc03.tar-split.gz (wrote 150320640 bytes)
Completed modify and upload for image docker.io/tripleomaster/centos-binary-mistral-executor:current-tripleo
Removing local copy of 192.168.24.1:8787/tripleomaster/centos-binary-mistral-executor:current-tripleo
Removing local copy of 192.168.24.1:8787/tripleomaster/centos-binary-mistral-executor:current-tripleo-devel-ps3
Output env file exists, moving it to backup.

If I were deploying the mistral container in the overcloud I could just use 'openstack overcloud deploy ... -e ~/containers-env-file.yaml' and be done, but because I need to replace my mistral-executor container on my undercloud I have to do a few manual steps.

Run the patched container on the undercloud

My undercloud is ready to serve the patched mistral-executor container but it doesn't yet have its own copy of it to run; i.e. I only see the original container:


(undercloud) [stack@undercloud train]$ sudo podman images | grep exec
docker.io/tripleomaster/centos-binary-mistral-executor current-tripleo 1f0ed5edc023 9 days ago 1.78 GB
(undercloud) [stack@undercloud train]$
However, the same undercloud will serve it from the following URL:

(undercloud) [stack@undercloud train]$ grep executor ~/containers-env-file.yaml
ContainerMistralExecutorImage: 192.168.24.1:8787/tripleomaster/centos-binary-mistral-executor:current-tripleo-devel-ps3
(undercloud) [stack@undercloud train]$
So we can pull it down so we can run it on the undercloud:

sudo podman pull 192.168.24.1:8787/tripleomaster/centos-binary-mistral-executor:current-tripleo-devel-ps3
I now want to stop the running mistral-executor container and start my new one in it's place. As per Debugging with Paunch I can use the print-cmd action to extract the command which is used to start the mistral-executor container and save it to a shell script:

sudo paunch debug --file /var/lib/tripleo-config/container-startup-config-step_4.json --container mistral_executor --action print-cmd > start_executor.sh
I'll also add the exact container image name to the shell script

sudo podman images | grep ps3 >> start_executor.sh
Next I'll edit the script to update the container name and make sure the container is named mistral_executor:

vim start_executor.sh
Before I restart the container I'll prove that the current container isn't running the patch (the same command later will prove that it is).

(undercloud) [stack@undercloud train]$ sudo podman exec mistral_executor grep render /usr/lib/python2.7/site-packages/tripleo_common/utils/config.py
# string so it's rendered in a readable format.
template_data = deployment_template.render(
template_data = host_var_server_template.render(
(undercloud) [stack@undercloud train]$
Stop the mistral-executor container with systemd (otherwise it will automatically restart).

sudo systemctl stop tripleo_mistral_executor.service
Remove the container with podman to ensure the name is not in use:

sudo podman rm mistral_executor
Start the new container:

sudo bash start_executor.sh
and now I'll verify that my new container does have the patch:

(undercloud) [stack@undercloud train]$ sudo podman exec mistral_executor grep render /usr/lib/python2.7/site-packages/tripleo_common/utils/config.py
def render_network_config(self, stack, config_dir, server_roles):
# string so it's rendered in a readable format.
template_data = deployment_template.render(
template_data = host_var_server_template.render(
self.render_network_config(stack, config_dir, server_roles)
(undercloud) [stack@undercloud train]$
For a bonus, I also see it fixed the bug.

(undercloud) [stack@undercloud tripleo-heat-templates]$ openstack overcloud config download --config-dir config-download
Starting config-download export...
config-download export successful
Finished config-download export.
Extracting config-download...
The TripleO configuration has been successfully generated into: config-download
(undercloud) [stack@undercloud tripleo-heat-templates]$

by Unknown (noreply@blogger.com) at July 03, 2019 08:04 PM

June 21, 2019

Emilien Macchi

Developer workflow with TripleO

In this post we’ll see how one can use TripleO for developing & testing changes into OpenStack Python-based projects (e.g. Keystone).

 

Even if Devstack remains a popular tool, it is not the only one you can use for your development workflow.

TripleO hasn’t only been built for real-world deployments but also for developers working on OpenStack related projects like Keystone for example.

Let’s say, my Keystone directory where I’m writing code is in /home/emilien/git/openstack/keystone.

Now I want to deploy TripleO with that change and my code in Keystone. For that I will need a server (can be a VM) with at least 8GB of RAM, 4 vCPU and 50GB of disk, and CentOS7 or Fedora28 installed.

Prepare the repositories and install python-tripleoclient:

If you’re deploying on recent Fedora or RHEL8, you’ll need to install python3-tripleoclient.

Now, let’s prepare your environment and deploy TripleO:

Note: change the YAML for your own needs if needed. If you need more help on how to configure Standalone, please check out the official manual.

Now let’s say your code needs a change and you need to retest it. Once you modified your code, just run:

Now, if you need to test a review that is already pushed in Gerrit and you want to run a fresh deployment with it, you can do it with:

I hope these tips helped you to understand how you can develop and test any OpenStack Python-based project without pain, and pretty quickly. On my environment, the whole deployment takes less than 20 minutes.

Please give any feedback in comment or via email!

by Emilien at June 21, 2019 05:07 PM

June 17, 2019

Lars Kellogg-Stedman

Avoid rebase hell: squashing without rebasing

You’re working on a pull request. You’ve been working on a pull request for a while, and due to lack of sleep or inebriation you’ve been merging changes into your feature branch rather than rebasing. You now have a pull request that looks like this (I’ve marked merge commits with the text [merge]): 7e181479 Adds methods for widget sales 0487162 [merge] Merge remote-tracking branch 'origin/master' into my_feature 76ee81c [merge] Merge branch 'my_feature' of https://github.

June 17, 2019 12:00 AM

June 14, 2019

Lars Kellogg-Stedman

Git Etiquette: Commit messages and pull requests

Always work on a branch (never commit on master) When working with an upstream codebase, always make your changes on a feature branch rather than your local master branch. This will make it easier to keep your local master branch current with respect to upstream, and can help avoid situations in which you accidentally overwrite your local changes or introduce unnecessary merge commits into your history. Rebase instead of merge If you need to incorporate changes from the upstream master branch in the feature branch on which you are currently doing, bring in those changes using git rebase rather than git merge.

June 14, 2019 12:00 AM

June 07, 2019

Lars Kellogg-Stedman

Running Keystone with Docker Compose

In this article, we will look at what is necessary to run OpenStack’s Keystone service (and the requisite database server) in containers using Docker Compose. Running MariaDB The standard mariadb docker image can be configured via a number of environment variables. It also benefits from persistent volume storage, since in most situations you don’t want to lose your data when you remove a container. A simple docker command line for starting MariaDB might look something like:

June 07, 2019 12:00 AM

May 30, 2019

Website and blog of Pablo Iranzo Gómez

Emerging Tech VLC‘19 - Citellus - Automatización de comprobaciones

Citellus:

Citellus - Verifica tus sistemas!!

https://citellus.org

Emerging Tech Valencia 2019: 30 Mayo

¿Quién soy?

Involucrado con Linux desde algo antes de comenzar los estudios universitarios y luego durante ellos, estando involucrado con las asociaciones LinUV y <Valux.org>.

Empecé a ‘vivir’ del software libre en 2004 y a trabajar en Red Hat en 2006 como Consultor, luego como Senior Technical Account Manager, posteriormente como Principal Software Maintenance Engineer y actualmente como Senior Software Engineer para el equipo de Solutions Engineering.

¿Qué es Citellus?

  • Citellus es un framework acompañado de scripts creados por la comunidad, que automatizan la detección de problemas, incluyendo problemas de configuración, conflictos con paquetes de versiones instaladas, problemas de seguridad o configuraciones inseguras y mucho más.

Historia: ¿cómo comenzó el proyecto?

  • Un fin de semana de guardia revisando una y otra vez la mismas configuraciones en diversos equipos hicieron evidente la necesidad de automatizar.

  • Unos scripts sencillos y un ‘wrapper’ en bash después, la herramienta fue tomando forma, poco después, se reescribió el ‘wrapper’ en python para proporcionarle características más avanzadas.

  • En esos primeros momentos también se mantuvieron conversaciones con ingeniería y como resultado, un nuevo diseño de los tests más sencillo fue adoptado.

¿Qué puedo hacer con Citellus?

  • Ejecutarlo contra un sistema o contra un sosreport.
  • Resolver antes los problemas gracias a la información que proporciona.
  • Utilizar los plugins para detectar problemas actuales o futuros (ciclo de vida, etc).
  • Programar nuevos plugins en tu lenguaje de programación preferido (bash, python, ruby, etc.) para extender la funcionalidad.
    • Contribuir al proyecto esos nuevos plugins para beneficio de otros.
  • Utilizar dicha información como parte de acciones proactivas en sus sistemas.

¿Algún ejemplo de la vida real?

  • Por ejemplo, con Citellus puedes detectar:
    • Borrados incorrectos de tokens de keystone
    • Parámetros faltantes para expirar y purgar datos de ceilometer que pueden llevar a llenar el disco duro.
    • NTP no sincronizado
    • paquetes obsoletos que están afectados por fallos críticos o de seguridad.
    • otros! (860+) complementos en este momento, con más de una comprobación por plugin en muchos de ellos
  • Cualquier otra cosa que puedas imaginar o programar 😉

Cambios derivados de ejemplos reales?

  • Inicialmente trabajábamos con RHEL únicamente (6, 7 y 8) por ser las soportadas
  • Dado que trabajamos con otros equipos internos como RHOS-OPS que utilizan por ejemplo RDO project, la versión upstream de Red Hat OpenStack, comenzamos a adaptar tests para funcionar en ambas.
  • A mayores, empezamos a crear funciones adicionales para operar sobre sistemas Debian y un compañero estuvo también enviando propuestas para corregir algunos fallos sobre Arch Linux.
  • Con la aparición de Spectre y Meltdown empezamos a añadir también comprobación de algunos paquetes y que no se hayan deshabilitado las opciones para proteger frente a dichos ataques.

Algunos números sobre plugins:

- healthcheck : 79 [] - informative : 2 [] - negative : 3 [‘system: 1’, ‘system/iscsi: 1’] - openshift : 5 [] - openstack : 4 [‘rabbitmq: 1’] - ovirt-rhv : 1 [] - pacemaker : 2 [] - positive : 35 [‘cluster/cman: 1’, ‘openstack: 16’, ‘openstack/ceilometer: 1’, ‘system: 1’] - rhinternal : 697 [‘bugzilla/docker: 1’, ‘bugzilla/httpd: 1’, ‘bugzilla/openstack/ceilometer: 1’, ‘bugzilla/openstack/ceph: 1’, ‘bugzilla/openstack/cinder: 1’, ‘bugzilla/openstack/httpd: 1’, ‘bugzilla/openstack/keystone: 1’, ‘bugzilla/openstack/keystone/templates: 1’, ‘bugzilla/openstack/neutron: 5’, ‘bugzilla/openstack/nova: 4’, ‘bugzilla/openstack/swift: 1’, ‘bugzilla/openstack/tripleo: 2’, ‘bugzilla/systemd: 1’, ‘ceph: 4’, ‘cifs: 5’, ‘docker: 1’, ‘httpd: 1’, ‘launchpad/openstack/keystone: 1’, ‘launchpad/openstack/oslo.db: 1’, ‘network: 7’, ‘ocp-pssa/etcd: 1’, ‘ocp-pssa/master: 12’, ‘ocp-pssa/node: 14’, ‘openshift/cluster: 1’, ‘openshift/etcd: 2’, ‘openshift/node: 1’, ‘openshift/ocp-pssa/master: 2’, ‘openstack: 6’, ‘openstack/ceilometer: 2’, ‘openstack/ceph: 1’, ‘openstack/cinder: 5’, ‘openstack/containers: 4’, ‘openstack/containers/docker: 2’, ‘openstack/containers/rabbitmq: 1’, ‘openstack/crontab: 4’, ‘openstack/glance: 1’, ‘openstack/haproxy: 2’, ‘openstack/hardware: 1’, ‘openstack/iptables: 1’, ‘openstack/keystone: 3’, ‘openstack/mysql: 8’, ‘openstack/network: 6’, ‘openstack/neutron: 5’, ‘openstack/nova: 12’, ‘openstack/openvswitch: 3’, ‘openstack/pacemaker: 1’, ‘openstack/rabbitmq: 5’, ‘openstack/redis: 1’, ‘openstack/swift: 3’, ‘openstack/system: 4’, ‘openstack/systemd: 1’, ‘pacemaker: 10’, ‘satellite: 1’, ‘security: 3’, ‘security/meltdown: 2’, ‘security/spectre: 8’, ‘security/speculative-store-bypass: 8’, ‘storage: 1’, ‘sumsos/bugzilla: 11’, ‘sumsos/kbases: 426’, ‘supportability: 11’, ‘sysinfo: 2’, ‘system: 56’, ‘virtualization: 2’] - supportability : 3 [‘openshift: 1’] - sysinfo : 18 [‘lifecycle: 6’, ‘openshift: 4’, ‘openstack: 2’] - system : 12 [‘iscsi: 1’] - virtualization : 1 [] ——- total : 862

El Objetivo

  • Hacer extremadamente sencillo escribir nuevos plugins.
  • Permitir escribirlos en tu lenguaje de programación preferido.
  • Que sea abierto para que cualquiera pueda contribuir.

Cómo ejecutarlo?

A destacar

  • plugins en su lenguaje preferido
  • Permite sacar la salida a un fichero json para ser procesada por otras herramientas.
    • Permite visualizar via html el json generado
  • Soporte de playbooks ansible (en vivo y también contra un sosreport si se adaptan)
    • Las extensiones (core, ansible), permiten extender el tipo de plugins soportado fácilmente.
  • Salvar/restaurar la configuración
  • Instalar desde pip/pipsi si no quieres usar el git clone del repositorio o ejecutar desde un contenedor.

Interfaz HTML

  • Creado al usar –web, abriendo fichero citellus.html por http se visualiza.

¿Por qué upstream?

  • Citellus es un proyecto de código abierto. Todos los plugins se envían al repositorio en github para compartirlos (es lo que queremos fomentar, reutilización del conocimiento).
  • Cada uno es experto en su área: queremos que todos contribuyan
  • Utilizamos un acercamiento similar a otros proyectos de código abierto: usamos gerrit para revisar el código y UnitTesting para validar la funcionalidad básica.

¿Cómo contribuir?

Actualmente hay una gran presencia de plugins de OpenStack, ya que es en ese área donde trabajamos diariamente, pero Citellus no está limitado a una tecnología o producto.

Por ejemplo, es fácil realizar comprobaciones acerca de si un sistema está configurado correctamente para recibir actualizaciones, comprobar versiones específicas con fallos (Meltdown/Spectre) y que no hayan sido deshabilitadas las protecciones, consumo excesivo de memoria por algún proceso, fallos de autentificación, etc.

Lea la guía del colaborador en: https://github.com/citellusorg/citellus/blob/master/CONTRIBUTING.md para más detalles.

Citellus vs otras herramientas

  • XSOS: Proporciona información de datos del sistema (ram, red, etc), pero no analiza, a los efectos es un visor ‘bonito’ de información.

  • TripleO-validations: se ejecuta solamente en sistemas ‘en vivo’, poco práctico para realizar auditorías o dar soporte.

¿Por qué no sosreports?

  • No hay elección entre una u otra, SOS recoge datos del sistema, Citellus los analiza.
  • Sosreport viene en los canales base de RHEL, Debian que hacen que esté ampliamente distribuido, pero también, dificulta el recibir actualizaciones frecuentes.
  • Muchos de los datos para diagnóstico ya están en los sosreports, falta el análisis.
  • Citellus se basa en fallos conocidos y es fácilmente extensible, necesita ciclos de desarrollo más cortos, estando más orientado a equipos de devops o de soporte.

¿Qué hay bajo el capó?

Filosofía sencilla:

  • Citellus es el ‘wrapper’ que ejecuta.
  • Permite especificar la carpeta con el sosreport
  • Busca los plugins disponibles en el sistema
  • Lanza los plugins contra cada sosreport y devuelve el estado.
  • El framework de Citellus en python permite manejo de opciones, filtrado, ejecución paralela, etc.

¿Y los plugins?

Los plugins son aún más sencillos:

  • En cualquier lenguaje que pueda ser ejecutado desde una shell.
  • Mensajes de salida a ‘stderr’ (>&2)
  • Si en bash se utilizan cadenas como $“cadena”, se puede usar el soporte incluido de i18n para traducirlos al idioma que se quiera.
  • Devuelve $RC_OKAY si el test es satisfactorio / $RC_FAILED para error / $RC_SKIPPED para los omitidos / Otro para fallos no esperados.

¿Y los plugins? (continuación)

  • Heredan variables del entorno como la carpeta raíz para el sosreport (vacía en modo Live) (CITELLUS_ROOT) o si se está ejecutando en modo live (CITELLUS_LIVE). No se necesita introducir datos vía el teclado
  • Por ejemplo los tests en ‘vivo’ pueden consultar valores en la base de datos y los basados en sosreport, limitarse a los logs existentes.

Ejemplo de script

¿Listos para profundizar en los plugins?

  • Cada plugin debe validar si debe o no ejecutarse y mostrar la salida a ‘stderr’, código de retorno.
  • Citellus ejecutará e informará de los tests en base a los filtros usados.

Requisitos:

  • El código de retorno debe ser $RC_OKAY (ok), $RC_FAILED (fallo) or $RC_SKIPPED (omitido).
  • Los mensajes impresos a stderr se muestran si el plugin falla o se omite (si se usa el modo detallado)
  • Si se ejecuta contra un ‘sosreport’, la variable CITELLUS_ROOT tiene la ruta a la carpeta del sosreport indicada.
  • CITELLUS_LIVE contiene 0 ó 1 si es una ejecución en vivo o no.

¿Cómo empezar un nuevo plugin (por ejemplo)?

  • Crea un script en ~/~/.../plugins/core/rhev/hosted-engine.sh
  • chmod +x hosted-engine.sh

¿Cómo empezar un nuevo plugin (continuación)?

¿Cómo empezar un nuevo plugin (con funciones)?

¿Cómo probar un plugin?

  • Use tox para ejecutar algunas pruebas UT (utf8, bashate, python 2.7, python 3)

  • Diga a Citellus qué plugin utilizar:

¿Qué es Magui?

Introducción

  • Citellus trabaja a nivel de sosreport individual, pero algunos problemas se manifiestan entre conjuntos de equipos (clústeres, virtualización, granjas, etc)

Por ejemplo, Galera debe comprobar el seqno entre los diversos miembros para ver cúal es el que contiene los datos más actualizados.

Qué hace M.a.g.u.i. ?

  • Ejecuta citellus contra cada sosreport o sistema, obtiene los datos y los agrupa por plugin.
  • Ejecuta sus propios plugins contra los datos obtenidos, destacando problemas que afectan al conjunto.
  • Permite obtener datos de equipos remotos via ansible-playbook.

¿Qué aspecto tiene?

Siguientes pasos con Magui?

  • Dispone de algunos plugins en este momento:
    • Agregan data de citellus ordenada por plugin para comparar rápidamente
    • Muestra los datos de ‘metadatos’ de forma separada para contrastar valores
    • pipeline-yaml, policy.json y otros (asociados a OpenStack)
    • seqno de galera
    • redhat-release entre equipos
    • Faraday: compara ficheros que deban ser iguales o distintos entre equipos

Siguientes pasos

  • Más plugins!
  • Dar a conocer la herramienta para entre todos, facilitar la resolución de problemas, detección de fallos de seguridad, configuraciones incorrectas, etc.
  • Movimiento: Muchas herramientas mueren por tener un único desarrollador trabajando en sus ratos libres, tener contribuciones es básico para cualquier proyecto.
  • Programar más tests en Magui para identificar más casos dónde los problemas aparecen a nivel de grupos de sistemas y no a nivel de sistema sindividuales.

Otros recursos

Blog posts:

¿Preguntas?

Gracias por asistir!!

Ven a #citellus en Freenode, https://t.me/citellusUG en Telegram o contacta con nosotros:

Presentación disponible en:

https://iranzo.github.io

by Pablo Iranzo Gómez at May 30, 2019 05:30 PM

May 21, 2019

Carlos Camacho

The Kubernetes in a box project

Implementing cloud computing solutions that runs in hybrid environments might be the final solution when comes to finding the best benefits/cost ratio.

This post will be the main thread to build and describe the KIAB/Kubebox project (www.kubebox.org and/or www.kiab.org).

Spoiler alert!

The name

First thing first, the name.. I have in my mind two names having the same meaning. The first one is KIAB (Kubernetes In A Box) this name came to my mind as the Kiai sound from karatekas (practitioners of karate). The second one is more traditional, “Kubebox”. I have no preference but it would be awesome if you help me to decide the official name for this project.

Add a comment and contribute to select the project name!

Introduction

This project is about to integrate together already market available devices to run cloud software as an appliance.

The proof-of-concept delivered in this series of posts will allow people to put a well-known set of hardware devices into a single chassis for either create their own cloud appliances, research and development, continuous integration, testing, home labs, staging or production-ready environments or simply just for fun.

Hereby it’s humbly presented to you the design of KubeBox/KIAB an open chassis specification for building cloud appliances.

The case enclosure is fully designed, and hopefully in the last phases for building the first set of enclosures, now, the posts will appear in the mean time I have some free cycles for writing the overall description.

Use cases

Several use cases can be defined to run on a KubeBox chassis.

  • AWS outpost.
  • Development environments.
  • EDGE.
  • Production Environments for small sites.
  • GitLab CI integration.
  • Demos for summits and conferences.
  • R&D: FPGA usage, deep learning, AI, TensorFlow, among many others.
  • Marketing WOW effect.
  • Training.

Enclosure design

The enclosure is designed as a rackable unit, using 7U. It tries to minimize the space used to deploy an up to 8-node cluster with redundancy for both power and networking.

Cloud appliance description

This build will be described across several sub-posts linked from this main thread. The posts will be created particularly without any specific order depending on my availability.

  • Backstory and initial parts selection.
  • Designing the case part 1: Design software.
  • A brief introduction to CAD software.
  • Designing the case part 2: U’s, brakes, and ghosts.
  • Designing the case part 3: Sheet thickness and bend radius.
  • Designing the case part 4: Parts Allowance (finish, tolerance, and fit).
  • Designing the case part 5: Vent cutouts and frickin’ laser beams!.
  • Designing the case part 6: Self-clinching nuts and standoffs.
  • Designing the case part 7: The standoffs strike back.
  • A brief primer on screws and PEMSERTs.
  • Designing the case part 8: Implementing PEMSERTs and screws.
  • Designing the case part 9: Bend reliefs and flat patterns.
  • Designing the case part 10: Tray caddy, to be used with GPU, Mother boards, disks, any other peripherals you want to add to the enclosure.
  • Designing the case part 11: Components rig.
  • Designing the case part 12: Power supply.
  • Designing the case part 13: Networking.
  • Designing the case part 14: 3D printed supports.
  • Designing the case part 15: Adding computing power.
  • Designing the case part 16: Adding Storage.
  • Designing the case part 17: Front display and bastion for automation.
  • Manufacturing the case part 1: PEMSERT installation.
  • Manufacturing the case part 2: Bending metal.
  • Manufacturing the case part 3: Bending metal.
  • KubeBox cloud appliance in detail!.
  • Manufacturing the case part 0: Getting quotes.
  • Manufacturing the case part 1: Getting the cases.
  • Software deployments: Reference architecture.
  • Design final source files for the enclosure design.
  • KubeBox is fully functional.

Update log:

2019/05/21: Initial version.

by Carlos Camacho at May 21, 2019 12:00 AM

May 20, 2019

Carlos Camacho

Running Relax-and-Recover to save your OpenStack deployment

ReaR is a pretty impressive disaster recovery solution for Linux. Relax-and-Recover, creates both a bootable rescue image and a backup of the associated files you choose.

When doing disaster recovery of a system, this Rescue Image plays the files back from the backup and so in the twinkling of an eye the latest state.

Various configuration options are available for the rescue image. For example, slim ISO files, USB sticks or even images for PXE servers are generated. As many backup options are possible. Starting with a simple archive file (eg * .tar.gz), various backup technologies such as IBM Tivoli Storage Manager (TSM), EMC NetWorker (Legato), Bacula or even Bareos can be addressed.

The ReaR written in Bash enables the skilful distribution of Rescue Image and if necessary archive file via NFS, CIFS (SMB) or another transport method in the network. The actual recovery process then takes place via this transport route.

In this specific case, due to the nature of the OpenStack deployment we will choose those protocols that are allowed by default in the Iptables rules (SSH, SFTP in particular).

But enough with the theory, here’s a practical example of one of many possible configurations. We will apply this specific use of ReaR to recover a failed control plane after a critical maintenance task (like an upgrade).

01 - Prepare the Undercloud backup bucket.

We need to prepare the place to store the backups from the Overcloud. From the Undercloud, check you have enough space to make the backups and prepare the environment. We will also create a user in the Undercloud with no shell access to be able to push the backups from the controllers or the compute nodes.

groupadd backup
mkdir /data
useradd -m -g backup -d /data/backup backup
echo "backup:backup" | chpasswd
chown -R backup:backup /data
chmod -R 755 /data

02 - Run the backup from the Overcloud nodes.

Let’s install some required packages and run some previous configuration steps.

#Install packages
sudo yum install rear genisoimage syslinux lftp wget -y

#Make sure you are able to use sshfs to store the ReaR backup
sudo yum install fuse -y
sudo yum groupinstall "Development tools" -y
wget http://download-ib01.fedoraproject.org/pub/epel/7/x86_64/Packages/f/fuse-sshfs-2.10-1.el7.x86_64.rpm
sudo rpm -i fuse-sshfs-2.10-1.el7.x86_64.rpm

sudo mkdir -p /data/backup
sudo sshfs -o allow_other backup@undercloud-0:/data/backup /data/backup
#Use backup password, which is... backup

Now, let’s configure ReaR config file.

#Configure ReaR
sudo tee -a "/etc/rear/local.conf" > /dev/null <<'EOF'
OUTPUT=ISO
OUTPUT_URL=sftp://backup:backup@undercloud-0/data/backup/
BACKUP=NETFS
BACKUP_URL=sshfs://backup@undercloud-0/data/backup/
BACKUP_PROG_COMPRESS_OPTIONS=( --gzip )
BACKUP_PROG_COMPRESS_SUFFIX=".gz"
BACKUP_PROG_EXCLUDE=( '/tmp/*' '/data/*' )
EOF

Now run the backup, this should create an ISO image in the Undercloud node (/data/backup/).

You will be asked for the backup user password

sudo rear -d -v mkbackup

Now, simulate a failure xD

# sudo rm -rf /lib

After the ISO image is created, we can proceed to verify we can restore it from the Hypervisor.

03 - Prepare the hypervisor.

# Enable the use of fusefs for the VMs on the hypervisor
setsebool -P virt_use_fusefs 1

# Install some required packages
sudo yum install -y fuse-sshfs

# Mount the Undercloud backup folder to access the images
mkdir -p /data/backup
sudo sshfs -o allow_other root@undercloud-0:/data/backup /data/backup
ls /data/backup/*

04 - Stop the damaged controller node.

virsh shutdown controller-0
# virsh destroy controller-0

# Wait until is down
watch virsh list --all

# Backup the guest definition
virsh dumpxml controller-0 > controller-0.xml
cp controller-0.xml controller-0.xml.bak

Now, we need to change the guest definition to boot from the ISO file.

Edit controller-0.xml and update it to boot from the ISO file.

Find the OS section,add the cdrom device and enable the boot menu.

<os>
<boot dev='cdrom'/>
<boot dev='hd'/>
<bootmenu enable='yes'/>
</os>

Edit the devices section and add the CDROM.

<disk type='file' device='cdrom'>
<driver name='qemu' type='raw'/>
<source file='/data/backup/rear-controller-0.iso'/>
<target dev='hdc' bus='ide'/>
<readonly/>
<address type='drive' controller='0' bus='1' target='0' unit='0'/>
</disk>

Update the guest definition.

virsh define controller-0.xml

Restart and connect to the guest

virsh start controller-0
virsh console controller-0

You should be able to see the boot menu to start the recover process, select Recover controller-0 and follow the instructions.

Now, before proceeding to run the controller restore, it’s possible that the host undercloud-0 can’t be resolved, just:

echo "192.168.24.1 undercloud-0" >> /etc/hosts

Having resolved the Undercloud host, just follow the wizard, Relax And Recover :)

You yould see a message like:

Welcome to Relax-and-Recover. Run "rear recover" to restore your system !

RESCUE controller-0:~ # rear recover

The image restore should progress quickly.

Continue to see the restore evolution.

Now, each time you reboot the node will have the ISO file as the first boot option so it’s something we need to fix. In the mean time let’s check if the restore went fine.

Reboot the guest booting from the hard disk.

Now we can see that the guest VM started successfully.

Now we need to restore the guest to it’s original definition, so from the Hypervisor we need to restore the controller-0.xml.bak file we created.

#From the Hypervisor
virsh shutdown controller-0
watch virsh list --all
virsh define controller-0.xml.bak
virsh start controller-0

Enjoy.

Considerations:

  • Space.
  • Multiple protocols supported but we might then to update firewall rules, that’s why I prefered SFTP.
  • Network load when moving data.
  • Shutdown/Starting sequence for HA control plane.
  • Do we need to backup the data plane?
  • User workloads should be handled by a third party backup software.

Update log:

2019/05/20: Initial version.

2019/06/18: Appeared in OpenStack Superuser blog.

by Carlos Camacho at May 20, 2019 12:00 AM

May 16, 2019

Website and blog of Pablo Iranzo Gómez

Emerging Tech VLC‘19: Citellus - Automatización de comprobaciones

Citellus:

Citellus - Verifica tus sistemas!!

https://citellus.org

Emerging Tech Valencia 2019: 16 Mayo

¿Quién soy?

Involucrado con Linux desde algo antes de comenzar los estudios universitarios y luego durante ellos, estando involucrado con las asociaciones LinUV y Valux.org.

Empecé a ‘vivir’ del software libre en 2004 y a trabajar en Red Hat en 2006 como Consultor, luego como Technical Account Manager y ahora como Software Maintenance Engineer.

¿Qué es Citellus?

  • Citellus proporciona un framework acompañado de scripts creados por la comunidad, que automatizan la detección de problemas, incluyendo problemas de configuración, conflictos con paquetes de versiones instaladas, problemas de seguridad o configuraciones inseguras y mucho más.

Historia: ¿cómo comenzó el proyecto?

  • Un fin de semana de guardia revisando una y otra vez las mismas configuraciones en diversos hosts sembró la semilla.

  • Unos scripts sencillos y un ‘wrapper’ en bash después, la herramienta fue tomando forma, poco después, se reescribió el ‘wrapper’ en python para proporcionarle características más avanzadas.

  • En esos primeros momentos también se mantuvieron conversaciones con ingeniería y como resultado, un nuevo diseño de los tests más sencillo fue adoptado.

¿Qué puedo hacer con Citellus?

  • Ejecutarlo contra un sistema en vivo o contra un sosreport.
  • Resolver problemas antes gracias a la información que proporciona.
  • Utilizar los plugins para detectar problemas actuales o futuros.
  • Programar nuevos plugins en tu lenguaje de programación preferido (bash, python, ruby, etc.) para extender la funcionalidad.
    • Contribuir al proyecto esos nuevos plugins para beneficio de otros.
  • Utilizar dicha información como parte de acciones proactivas en sus sistemas.

¿Algún ejemplo de la vida real?

  • Por ejemplo, con Citellus puedes detectar:
    • Borrados incorrectos de tokens de keystone
    • Parámetros faltantes para expirar y purgar datos de ceilometer que pueden llevar a llenar el disco duro.
    • NTP no sincronizado
    • paquetes obsoletos que están afectados por fallos críticos o de seguridad.
    • otros! (850+) complementos en este momento, con más de una comprobación por plugin en muchos de ellos
  • Cualquier otra cosa que puedas imaginar o programar 😉

Cambios derivados de ejemplos reales?

  • Inicialmente trabajábamos con RHEL únicamente (6 y 7) por ser las soportadas
  • Dado que trabajamos con otros equipos internos como RHOS-OPS que utilizan por ejemplo RDO project, la versión upstream de Red Hat OpenStack, comenzamos a adaptar tests para funcionar en ambas.
  • A mayores, empezamos a crear funciones adicionales para operar sobre sistemas Debian y un compañero estuvo también enviando propuestas para corregir algunos fallos sobre Arch Linux.
  • Con la aparición de Spectre y Meltdown empezamos a añadir también comprobación de algunos paquetes y que no se hayan deshabilitado las opciones para proteger frente a dichos ataques.

Algunos números sobre plugins:

- healthcheck : 79 [] - informative : 2 [] - negative : 3 [‘system: 1’, ‘system/iscsi: 1’] - openshift : 5 [] - openstack : 4 [‘rabbitmq: 1’] - ovirt-rhv : 1 [] - pacemaker : 2 [] - positive : 35 [‘cluster/cman: 1’, ‘openstack: 16’, ‘openstack/ceilometer: 1’, ‘system: 1’] - rhinternal : 697 [‘bugzilla/docker: 1’, ‘bugzilla/httpd: 1’, ‘bugzilla/openstack/ceilometer: 1’, ‘bugzilla/openstack/ceph: 1’, ‘bugzilla/openstack/cinder: 1’, ‘bugzilla/openstack/httpd: 1’, ‘bugzilla/openstack/keystone: 1’, ‘bugzilla/openstack/keystone/templates: 1’, ‘bugzilla/openstack/neutron: 5’, ‘bugzilla/openstack/nova: 4’, ‘bugzilla/openstack/swift: 1’, ‘bugzilla/openstack/tripleo: 2’, ‘bugzilla/systemd: 1’, ‘ceph: 4’, ‘cifs: 5’, ‘docker: 1’, ‘httpd: 1’, ‘launchpad/openstack/keystone: 1’, ‘launchpad/openstack/oslo.db: 1’, ‘network: 7’, ‘ocp-pssa/etcd: 1’, ‘ocp-pssa/master: 12’, ‘ocp-pssa/node: 14’, ‘openshift/cluster: 1’, ‘openshift/etcd: 2’, ‘openshift/node: 1’, ‘openshift/ocp-pssa/master: 2’, ‘openstack: 6’, ‘openstack/ceilometer: 2’, ‘openstack/ceph: 1’, ‘openstack/cinder: 5’, ‘openstack/containers: 4’, ‘openstack/containers/docker: 2’, ‘openstack/containers/rabbitmq: 1’, ‘openstack/crontab: 4’, ‘openstack/glance: 1’, ‘openstack/haproxy: 2’, ‘openstack/hardware: 1’, ‘openstack/iptables: 1’, ‘openstack/keystone: 3’, ‘openstack/mysql: 8’, ‘openstack/network: 6’, ‘openstack/neutron: 5’, ‘openstack/nova: 12’, ‘openstack/openvswitch: 3’, ‘openstack/pacemaker: 1’, ‘openstack/rabbitmq: 5’, ‘openstack/redis: 1’, ‘openstack/swift: 3’, ‘openstack/system: 4’, ‘openstack/systemd: 1’, ‘pacemaker: 10’, ‘satellite: 1’, ‘security: 3’, ‘security/meltdown: 2’, ‘security/spectre: 8’, ‘security/speculative-store-bypass: 8’, ‘storage: 1’, ‘sumsos/bugzilla: 11’, ‘sumsos/kbases: 426’, ‘supportability: 11’, ‘sysinfo: 2’, ‘system: 56’, ‘virtualization: 2’] - supportability : 3 [‘openshift: 1’] - sysinfo : 18 [‘lifecycle: 6’, ‘openshift: 4’, ‘openstack: 2’] - system : 12 [‘iscsi: 1’] - virtualization : 1 [] ——- total : 862

El Objetivo

  • Hacer extremadamente sencillo escribir nuevos plugins.
  • Permitir escribirlos en tu lenguaje de programación preferido.
  • Que sea abierto para que cualquiera pueda contribuir.

Cómo ejecutarlo?

A destacar

  • plugins en su lenguaje preferido
  • Permite sacar la salida a un fichero json para ser procesada por otras herramientas.
    • Permite visualizar via html el json generado
  • Soporte de playbooks ansible (en vivo y también contra un sosreport si se adaptan)
    • Las extensiones (core, ansible), permiten extender el tipo de plugins soportado fácilmente.
  • Salvar/restaurar la configuración
  • Instalar desde pip/pipsi si no quieres usar el git clone del repositorio o ejecutar desde un contenedor.

Interfaz HTML

  • Creado al usar –web, abriendo fichero citellus.html por http se visualiza.

¿Por qué upstream?

  • Citellus es un proyecto de código abierto. Todos los plugins se envían al repositorio en github para compartirlos (es lo que queremos fomentar, reutilización del conocimiento).
  • Cada uno es experto en su área: queremos que todos contribuyan
  • Utilizamos un acercamiento similar a otros proyectos de código abierto: usamos gerrit para revisar el código y UnitTesting para validar la funcionalidad básica.

¿Cómo contribuir?

Actualmente hay una gran presencia de plugins de OpenStack, ya que es en ese área donde trabajamos diariamente, pero Citellus no está limitado a una tecnología o producto.

Por ejemplo, es fácil realizar comprobaciones acerca de si un sistema está configurado correctamente para recibir actualizaciones, comprobar versiones específicas con fallos (Meltdown/Spectre) y que no hayan sido deshabilitadas las protecciones, consumo excesivo de memoria por algún proceso, fallos de autenticación, etc.

Lea la guía del colaborador en : https://github.com/citellusorg/citellus/blob/master/CONTRIBUTING.md para más detalles.

Citellus vs otras herramientas

  • XSOS: Proporciona información de datos del sistema (ram, red, etc), pero no analiza, a los efectos es un visor ‘bonito’ de información.

  • TripleO-validations: se ejecuta solamente en sistemas ‘en vivo’, poco práctico para realizar auditorías o dar soporte.

¿Por qué no sosreports?

  • No hay elección entre una u otra, SOS recoge datos del sistema, Citellus los analiza.
  • Sosreport viene en los canales base de RHEL, Debian que hacen que esté ampliamente distribuido, pero también, dificulta el recibir actualizaciones frecuentes.
  • Muchos de los datos para diagnóstico ya están en los sosreports, falta el análisis.
  • Citellus se basa en fallos conocidos y es fácilmente extensible, necesita ciclos de desarrollo más cortos, estando más orientado a equipos de devops o de soporte.

¿Qué hay bajo el capó?

Filosofía sencilla:

  • Citellus es el ‘wrapper’ que ejecuta.
  • Permite especificar la carpeta con el sosreport
  • Busca los plugins disponibles en el sistema
  • Lanza los plugins contra cada sosreport y devuelve el estado.
  • El framework de Citellus en python permite manejo de opciones, filtrado, ejecución paralela, etc.

¿Y los plugins?

Los plugins son aún más sencillos:

  • En cualquier lenguaje que pueda ser ejecutado desde una shell.
  • Mensajes de salida a ‘stderr’ (>&2)
  • Si en bash se utilizan cadenas como $“cadena”, se puede usar el soporte incluido de i18n para traducirlos al idioma que se quiera.
  • Devuelve $RC_OKAY si el test es satisfactorio / $RC_FAILED para error / $RC_SKIPPED para los omitidos / Otro para fallos no esperados.

¿Y los plugins? (continuación)

  • Heredan variables del entorno como la carpeta raíz para el sosreport (vacía en modo Live) (CITELLUS_ROOT) o si se está ejecutando en modo live (CITELLUS_LIVE). No se necesita introducir datos vía el teclado
  • Por ejemplo los tests en ‘vivo’ pueden consultar valores en la base de datos y los basados en sosreport, limitarse a los logs existentes.

Ejemplo de script

¿Listos para profundizar en los plugins?

  • Cada plugin debe validar si debe o no ejecutarse y mostrar la salida a ‘stderr’, código de retorno.
  • Citellus ejecutará e informará de los tests en base a los filtros usados.

Requisitos:

  • El código de retorno debe ser $RC_OKAY (ok), $RC_FAILED (fallo) or $RC_SKIPPED (omitido).
  • Los mensajes impresos a stderr se muestran si el plugin falla o se omite (si se usa el modo detallado)
  • Si se ejecuta contra un ‘sosreport’, la variable CITELLUS_ROOT tiene la ruta a la carpeta del sosreport indicada.
  • CITELLUS_LIVE contiene 0 ó 1 si es una ejecución en vivo o no.

¿Cómo empezar un nuevo plugin (por ejemplo)?

  • Crea un script en ~/~/.../plugins/core/rhev/hosted-engine.sh
  • chmod +x hosted-engine.sh

¿Cómo empezar un nuevo plugin (continuación)?

¿Cómo empezar un nuevo plugin (con funciones)?

¿Cómo probar un plugin?

  • Use tox para ejecutar algunas pruebas UT (utf8, bashate, python 2.7, python 3)

  • Diga a Citellus qué plugin utilizar:

¿Qué es Magui?

Introducción

  • Citellus trabaja a nivel de sosreport individual, pero algunos problemas se manifiestan entre conjuntos de equipos (clústeres, virtualización, granjas, etc)

Por ejemplo, Galera debe comprobar el seqno entre los diversos miembros para ver cúal es el que contiene los datos más actualizados.

Qué hace M.a.g.u.i. ?

  • Ejecuta citellus contra cada sosreport o sistema, obtiene los datos y los agrupa por plugin.
  • Ejecuta sus propios plugins contra los datos obtenidos, destacando problemas que afectan al conjunto.
  • Permite obtener datos de equipos remotos via ansible-playbook.

¿Qué aspecto tiene?

Siguientes pasos con Magui?

  • Dispone de algunos plugins en este momento:
    • Agregan data de citellus ordenada por plugin para comparar rápidamente
    • Muestra los datos de ‘metadatos’ de forma separada para contrastar valores
    • pipeline-yaml, policy.json y otros (asociados a OpenStack)
    • seqno de galera
    • redhat-release entre equipos
    • Faraday: compara ficheros que deban ser iguales o distintos entre equipos

Siguientes pasos

  • Más plugins!
  • Dar a conocer la herramienta para entre todos, facilitar la resolución de problemas, detección de fallos de seguridad, configuraciones incorrectas, etc.
  • Movimiento: Muchas herramientas mueren por tener un único desarrollador trabajando en sus ratos libres, tener contribuciones es básico para cualquier proyecto.
  • Programar más tests en Magui para identificar más casos dónde los problemas aparecen a nivel de grupos de sistemas y no a nivel de sistema sindividuales.

Otros recursos

Blog posts:

¿Preguntas?

Gracias por asistir!!

Ven a #citellus en Freenode, https://t.me/citellusUG en Telegram o contacta con nosotros:

by Pablo Iranzo Gómez at May 16, 2019 05:30 PM

May 14, 2019

Lars Kellogg-Stedman

A DIY CPAP Battery Box

A year or so ago I was diagnosed with sleep apnea, and since them I’ve been sleeping with a CPAP. This past weekend, I joined my daughter on a scout camping trip to a campground without readily accessible electricity. This would be the first time I found myself in this situation, and as the date approached, I realized I was going to have to build or buy some sort of battery solution for my CPAP.

May 14, 2019 12:00 AM

May 07, 2019

Lars Kellogg-Stedman

Unpacking a Python regular expression

I recently answered a question from Harsha Nalore on StackOverflow that involved using Ansible to extract the output of a command sent to a BigIP device of some sort. My solution – which I claim to be functional, but probably not optimal – involved writing an Ansible filter module to parse the output. That filter made use of a complex-looking regular expression. Harsha asked for some details on that regular expression works, and the existing StackOverflow answer didn’t really seem the write place for that: so, here we are.

May 07, 2019 10:00 AM

New comment system

As long as I’m switching site generators, it seems like a good idea to refresh the comment system as well. I’ve been using Disqus for a while, since when I started it was one of the only games in town. There are now alternatives of different sorts, and one in particular caught my eye: Utterances uses GitHub issues for storing comments, which seems like a fantastic idea. That means that comments will finally be stored in the same place as the blog content, which I think is a happy state of affairs.

May 07, 2019 09:00 AM

May 06, 2019

Lars Kellogg-Stedman

New static site generator

I’ve switched my static site generator from Pelican to Hugo. I’ve tried to ensure that all the old links continue to work correctly, but if you notice anything missing or otherwise not working as intended, please let me know by opening an issue. Thanks!

May 06, 2019 12:00 AM

April 26, 2019

RDO Blog

RDO Stein Released

The RDO community is pleased to announce the general availability of the RDO build for OpenStack Stein for RPM-based distributions, CentOS Linux and Red Hat Enterprise Linux. RDO is suitable for building private, public, and hybrid clouds. Stein is the 19th release from the OpenStack project, which is the work of more than 1200 contributors from around the world.

The release is already available on the CentOS mirror network at http://mirror.centos.org/centos/7/cloud/x86_64/openstack-stein/.

The RDO community project curates, packages, builds, tests and maintains a complete OpenStack component set for RHEL and CentOS Linux and is a member of the CentOS Cloud Infrastructure SIG. The Cloud Infrastructure SIG focuses on delivering a great user experience for CentOS Linux users looking to build and maintain their own on-premise, public or hybrid clouds.

All work on RDO and on the downstream release, Red Hat OpenStack Platform, is 100% open source, with all code changes going upstream first.

Photo by Yucel Moran on Unsplash

New and Improved

Interesting things in the Stein release include:

  • Ceph Nautilus is the default version of Ceph, a free-software storage platform, implements object storage on a single distributed computer cluster, and provides interfaces for object-, block- and file-level storage, within RDO (or is it the default without OpenStack?). Within Nautilus, the Ceph Dashboard has gained a lot of new functionality like support for multiple users / roles, SSO (SAMLv2) for user authentication, auditing support, a new landing page showing more metrics and health info, I18N support, and REST API documentation with Swagger API.

  • The extracted Placement service, used to track cloud resource inventories and usages to help other services effectively manage and allocate their resources, is now packaged as part of RDO. Placement has added the ability to target a candidate resource provider, easing specifying a host for workload migration, increased API performance by 50% for common scheduling operations, and simplified the code by removing unneeded complexity, easing future maintenance.

Other improvements include:

  • The TripleO deployment service, used to develop and maintain tooling and infrastructure able to deploy OpenStack in production, using OpenStack itself wherever possible, added support for podman and buildah for containers and container images. Open Virtual Network (OVN) is now the default network configuration and TripleO now has improved composable network support for creating L3 routed networks and IPV6 network support.

Contributors

During the Stein cycle, we saw the following new RDO contributors:

  • Sławek Kapłoński
  • Tobias Urdin
  • Lee Yarwood
  • Quique Llorente
  • Arx Cruz
  • Natal Ngétal
  • Sorin Sbarnea
  • Aditya Vaja
  • Panda
  • Spyros Trigazis
  • Cyril Roelandt
  • Pranali Deore
  • Grzegorz Grasza
  • Adam Kimball
  • Brian Rosmaita
  • Miguel Duarte Barroso
  • Gauvain Pocentek
  • Akhila Kishore
  • Martin Mágr
  • Michele Baldessari
  • Chuck Short
  • Gorka Eguileor

Welcome to all of you and Thank You So Much for participating!

But we wouldn’t want to overlook anyone. A super massive Thank You to all 74 contributors who participated in producing this release. This list includes commits to rdo-packages and rdo-infra repositories:

  • yatin
  • Sagi Shnaidman
  • Wes Hayutin
  • Rlandy
  • Javier Peña
  • Alfredo Moralejo
  • Bogdan Dobrelya
  • Sławek Kapłoński
  • Alex Schultz
  • Emilien Macchi
  • Lon
  • Jon Schlueter
  • Luigi Toscano
  • Eric Harney
  • Tobias Urdin
  • Chandan Kumar
  • Nate Johnston
  • Lee Yarwood
  • rabi
  • Quique Llorente
  • Chandan Kumar
  • Luka Peschke
  • Carlos Goncalves
  • Arx Cruz
  • Kashyap Chamarthy
  • Cédric Jeanneret
  • Victoria Martinez de la Cruz
  • Bernard Cafarelli
  • Natal Ngétal
  • hjensas
  • Tristan de Cacqueray
  • Marc Dequènes (Duck)
  • Juan Antonio Osorio Robles
  • Sorin Sbarnea
  • Rafael Folco
  • Nicolas Hicher
  • Michael Turek
  • Matthias Runge
  • Giulio Fidente
  • Juan Badia Payno
  • Zoltan Caplovic
  • agopi
  • marios
  • Ilya Etingof
  • Steve Baker
  • Aditya Vaja
  • Panda
  • Florian Fuchs
  • Martin André
  • Dmitry Tantsur
  • Sylvain Baubeau
  • Jakub Ružička
  • Dan Radez
  • Honza Pokorny
  • Spyros Trigazis
  • Cyril Roelandt
  • Pranali Deore
  • Grzegorz Grasza
  • Bnemec
  • Adam Kimball
  • Haikel Guemar
  • Daniel Mellado
  • Bob Fournier
  • Nmagnezi
  • Brian Rosmaita
  • Ade Lee
  • Miguel Duarte Barroso
  • Alan Bishop
  • Gauvain Pocentek
  • Akhila Kishore
  • Martin Mágr
  • Michele Baldessari
  • Chuck Short
  • Gorka Eguileor

The Next Release Cycle

At the end of one release, focus shifts immediately to the next, Train, which has an estimated GA the week of 14-18 October 2019. The full schedule is available at https://releases.openstack.org/train/schedule.html.

Twice during each release cycle, RDO hosts official Test Days shortly after the first and third milestones; therefore, the upcoming test days are 13-14 June 2019 for Milestone One and 16-20 September 2019 for Milestone Three.

Get Started

There are three ways to get started with RDO.

To spin up a proof of concept cloud, quickly, and on limited hardware, try an All-In-One Packstack installation. You can run RDO on a single node to get a feel for how it works.

For a production deployment of RDO, use the TripleO Quickstart and you’ll be running a production cloud in short order.

Finally, for those that don’t have any hardware or physical resources, there’s the OpenStack Global Passport Program. This is a collaborative effort between OpenStack public cloud providers to let you experience the freedom, performance and interoperability of open source infrastructure. You can quickly and easily gain access to OpenStack infrastructure via trial programs from participating OpenStack public cloud providers around the world.

Get Help

The RDO Project participates in a Q&A service at https://ask.openstack.org. We also have our users@lists.rdoproject.org for RDO-specific users and operrators. For more developer-oriented content we recommend joining the dev@lists.rdoproject.org mailing list. Remember to post a brief introduction about yourself and your RDO story. The mailing lists archives are all available at https://mail.rdoproject.org. You can also find extensive documentation on RDOproject.org.

The #rdo channel on Freenode IRC is also an excellent place to find and give help.

We also welcome comments and requests on the CentOS mailing lists and the CentOS and TripleO IRC channels (#centos, #centos-devel, and #tripleo on irc.freenode.net), however we have a more focused audience within the RDO venues.

Get Involved

To get involved in the OpenStack RPM packaging effort, check out the RDO contribute pages, peruse the CentOS Cloud SIG page, and inhale the RDO packaging documentation.

Join us in #rdo on the Freenode IRC network and follow us on Twitter @RDOCommunity. You can also find us on Facebook and YouTube.

by Rain Leander at April 26, 2019 11:03 PM

Lars Kellogg-Stedman

Adding support for privilege escalation to Ansible's docker connection driver

I often use Docker to test out Ansible playbooks. While normally that works great, I recently ran into an unexpected problem with privilege escalation. Given a simple playbook like this:

---
- hosts: all
  gather_facts: false
  become: true
  tasks:
    - ping:

And an inventory like this:

all:
  vars:
    ansible_user: example
    ansible_connection: docker
  hosts …

by Lars Kellogg-Stedman at April 26, 2019 04:00 AM

Adding support for privilege escalation to Ansible's docker connection driver

Update 2019-05-09 Pull request #55816 has merged, so you can now use sudo with the docker connection driver even when sudo is configured to require a password. I often use Docker to test out Ansible playbooks. While normally that works great, I recently ran into an unexpected problem with privilege escalation. Given a simple playbook like this: --- - hosts: all gather_facts: false become: true tasks: - ping: And an inventory like this:

April 26, 2019 12:00 AM

April 25, 2019

Lars Kellogg-Stedman

Writing Ansible filter plugins

I often see questions from people who are attemping to perform complex text transformations in their Ansible playbooks. While I am a huge fan of Ansible, data transformation is not one of its strong points. For example, this past week someone asked a question on Stack Overflow in which they …

by Lars Kellogg-Stedman at April 25, 2019 04:00 AM

Writing Ansible filter plugins

I often see questions from people who are attemping to perform complex text transformations in their Ansible playbooks. While I am a huge fan of Ansible, data transformation is not one of its strong points. For example, this past week someone asked a question on Stack Overflow in which they were attempting to convert the output of the keytool command into a list of dictionaries. The output of the keytool -list -v command looks something like this:

April 25, 2019 12:00 AM

April 18, 2019

Emilien Macchi

Day 2 operations in OpenStack TripleO (episode 1: scale-down)

Scale-up and scale-down are probably the most common operations done after the initial deployment. Let’s see how they are getting improved. This first episode is about scale-down precisely.

How it works now

Right now when an operator runs “openstack overcloud node delete” command, it’ll update the Heat stack to remove the resources associated to the node(s) that we delete. It can be problematic for some services like Nova, Neutron and the Subscription Manager, which needs to be teared down before the server is deleted.

Proposal

The idea is to create an interface where we can run Ansible tasks which will be executed during the scale-down, before the nodes get deleted by Heat. The Ansible tasks will live near to the deployment / upgrade / … tasks that are in TripleO Heat Templates. Here is an example with Red Hat Subscription Management:

It involves 3 changes:

What’s next?

  • Getting reviews & feedback on the 3 patches
  • Implement scale down tasks for Neutron, Nova and Ceph, waiting for this feature
  • Looking at scale-up tasks

Demo 1

This demo shows a node being unsubscribed when the overcloud is scaled down.

Demo 2

This demo show a compute node being removed from the Overcloud.

by Emilien at April 18, 2019 03:21 PM

March 14, 2019

Adam Young

Building the Kolla Keystone Container

Kolla has become the primary source of Containers for running OpenStack services. Since if has been a while since I tried deliberately running just the Keystone container, I decided to build the Kolla version from scratch and run it.

UPDATE: Ozz wrote it already, and did it better: http://jaormx.github.io/2017/testing-containerized-openstack-services-with-kolla/

I had an clone of the Kolla repo already, but if you need one, you can get it by cloning

git clone git://git.openstack.org/openstack/kolla

All of the dependencies you need to run the build process are handled by tox. Assuming you can run tox elsewhere, you can use that here, too:

tox -e py35

That will run through all the unit tests. They do not take that long.

To build all of the containers you can active the virtual environment and then use the build tool. That takes quite a while, since there are a lot of containers required to run OpenStack.

$ . .tox/py35/bin/activate
(py35) [ayoung@ayoungP40 kolla]$ tools/build.py 

If you want to build just the keystone containers….

 python tools/build.py keystone

Building this with no base containers cached took me 5 minutes. Delta builds should be much faster.

Once the build is complete, you will have a bunch of container images defined on your system:

REPOSITORY TAG IMAGE ID CREATEDSIZE
kolla/centos-binary-keystone 7.0.2 69049739bad6 33 minutes ago 800 MB
kolla/centos-binary-keystone-fernet 7.0.2 89977265fcbb 33 minutes ago 800 MB
kolla/centos-binary-keystone-ssh 7.0.2 4b377e854980 33 minutes ago 819 MB
kolla/centos-binary-barbican-keystone-listener 7.0.2 6265d0acff16 33 minutes ago 732 MB
kolla/centos-binary-keystone-base 7.0.2 b6d78b9e0769 33 minutes ago 774 MB
kolla/centos-binary-barbican-base 7.0.2 ccd7b4ff311f 34 minutes ago 706 MB
kolla/centos-binary-openstack-base 7.0.2 38dbb3c57448 34 minutes ago 671 MB
kolla/centos-binary-base 7.0.2 177c786e9b01 36 minutes ago 419 MB
docker.io/centos 7 1e1148e4cc2c 3 months ago 202 MB

Note that the build instructions live in the git repo under docs.

by Adam Young at March 14, 2019 03:43 PM

February 24, 2019

Lars Kellogg-Stedman

Docker build learns about secrets and ssh agent forwarding

A common problem for folks working with Docker is accessing resources which require authentication during the image build step. A particularly common use case is getting access to private git repositories using ssh key-based authentication. Until recently there hasn't been a great solution:

  • you can embed secrets in your image …

by Lars Kellogg-Stedman at February 24, 2019 05:00 AM

Docker build learns about secrets and ssh agent forwarding

A common problem for folks working with Docker is accessing resources which require authentication during the image build step. A particularly common use case is getting access to private git repositories using ssh key-based authentication. Until recently there hasn’t been a great solution: you can embed secrets in your image, but now you can’t share the image with anybody. you can use build arguments, but this requires passing in an unenecrypted private key on the docker build command line, which is suboptimal for a number of reasons you can perform all the steps requiring authentication at runtime, but this can needlessly complicate your container startup process.

February 24, 2019 12:00 AM

February 12, 2019

Emilien Macchi

OpenStack Containerization with Podman – Part 5 (Image Build)

For this fifth episode, we’ll explain how we will build containers with Buildah. Don’t miss the first, secondthird and fourth episodes where we learnt how to deploy, operate, upgrade and monitor Podman containers.

In this post, we’ll see the work that we can replace Docker by Buildah to build our container images.

Context

In OpenStack TripleO, we have nearly 150 images (all layers included) for all the services that we can deploy. Of course you don’t need to build them all when deploying your OpenStack cloud, but in our production chain we build them all and push the images to a container registry, consumable by the community.

Historically, we have been using “kolla-build” and the process to leverage the TripleO images build is documented here.

Implementation

kolla-build only supports Docker CLI at this time and we recognized that changing its code to support something else sounded a painful plan, as Docker was hardcoded almost everywhere.

We decided to leverage kolla-build to generate the templates of the images, which is actually a tree of Dockerfile per container.

The dependencies format generated by Kolla is a JSON:

So what we do is that when running:

openstack overcloud container image build --use-buildah

We will call kolla-build with –list-dependencies that generates a directory per image, where we have a Dockerfile + other things needed during the builds.

Anyway, bottom line is: we still use Kolla to generate our templates but don’t want Docker to actually build the images.

In tripleo-common, we are implementing a build and push that will leverage “buildah bud” and “buildah push”.

buildah bud” is a good fit for us because it allows us to use the same logic and format as before with Docker (bud == build-using-dockerfile).

The main challenge for us is that our images aren’t small, and we have a lot of images to build, in our production chain. So we decided to parallelize the last layers of the images (which don’t have childs).

For example, 2 images at the same layer level will be built together, also a child won’t be built in parallel of its parent layer.

Here is a snippet of the code that will take the dependencies dictionary and build our containers:

Without the “fat daemon” that is Docker, using Buildah puts some challenges here where running multiple builds at the same time can be slow because of the locks to avoid race conditions and database corruptions. So we capped the number of workers to 8, to not make Buildah locking too hard on the system.

What about performances? This question is still under investigation. We are still testing our code and measuring how much time it takes to build our images with Buildah. One thing is sure, you don’t want to use vfs storage backend and use overlayfs. To do so, you’ll need to run at least Fedora 28 with 4.18 kernel, install fuse-overlayfs and Buildah should use this backend by default.

Demo

Please select full screen:

In the next episode, we’ll see how we are replacing the docker-registry by a simple web server. Stay tuned!

by Emilien at February 12, 2019 03:22 AM

February 11, 2019

RDO Blog

python-tempestconf's journey

For those who are not familiar with the python-tempestconf, it’s a tool for generating a tempest configuration file, which is required for running Tempest tests against a live OpenStack cluster. It queries a cloud and automatically discovers cloud settings, which weren’t provided by a user.

Internal project

In August 2016 config_tempest tool was decoupled from Red Hat Tempest fork and the python-tempestconf repository under the github redhat-openstack organization was created. The tool became an internal tool used for generating tempest.conf in downstream jobs which were running Tempest.

Why we like `python-tempestconf`

The reason why is quite easy. We at Red Hat were (and still are) running many different OpenStack jobs with different configurations which execute Tempest. And there python-tempestconf stepped in. We didn’t have to implement the logic for creating or modifying tempest.conf within the job configuration, we just used python-tempestconf which did that for us. It’s not only about the generating tempest.conf itself, because the tool also creates basic users, uploads an image and creates basic flavors which all of them are required for running Tempest tests.

Usage of python-tempestconf was also beneficial for engineers who liked the idea of not struggling with creating a tempest.conf file from scratch but rather using the tool which was able to generate it for them. The generated tempest.conf was sufficient for running simple Tempest tests.

Imagine you have a fresh OpenStack deployment and you want to run some Tempest tests, because you want to make sure that the deployment was successful. In order to do that, you can run the python-tempestconf which will do the basic configuration for you and will generate a tempest.conf, and execute Tempest. That’s it, isn’t it easy?

I have to admit, when I joined Red Hat and more specifically OpenStack team, I kind of struggled with all the information about OpenStack and Tempest, it was too much new information. Therefore I really liked when I could generate a tempest.conf which I could use for running just basic tests. If I had to generate the tempest.conf myself, my learning process would be a little bit slower. Therefore, I’m really grateful that we had the tool at that time.

Shipping in a package

At the beginning of 2017 we started to ship python-tempestconf rpm package. It’s available in RDO repositories from Ocata and higher. python-tempestconf package is also installed as a dependency of openstack-tempest package. So if a user installs openstack-tempest, also python-tempestconf will be installed. At this time, we also changed the entrypoint and the tool is executed via discover-tempest-config command. However, you could have already read all about it in this article.

Upstream project

By the end of 2017 python-tempestconf became an upstream project and got under OpenStack organization.

We have significantly improved the tool since then, not only its code but also its documentation, which contains all the required information for a user, see here. In my opinion every project which is designed for wider audience of users (python-tempestconf is an upstream project, so this condition is fulfilled), should have a proper documentation. Following python-tempestconf’s documentation should be any user able to execute it, set wanted arguments and set some special tempest options without any bigger problems.

I would say that there are 3 greatest improvements. One of them is the user documentation, which I’ve already mentioned. The second and third are improvements of the code itself and they are os-client-config integration and refactoring of the code in order to simplify adding new OpenStack services the tool can generate config for.

os-client-config is a library for collecting client configuration for using an OpenStack cloud in a consistent way. By importing the library a user can specify OpenStack credentials by 2 different ways:

  • Using OS_* environment variables, which is maybe the most common way. It requires sourcing credentials before running python-tempestconf. In case of packstack environment, it’s keystonerc_admin/demo file and in case of devstack there is openrc script.
  • Using --os-cloud parameter which takes one argument – name of the cloud which holds the required credentials. Those are stored in a cloud.yaml file.

The second code improvement was the simplification of adding new OpenStack services the tool can generate tempest.conf for. If you want to add a service, just create a bug in our storyboard, see python-tempestconf’s contributor guide. If you feel like it, you can also implement it. Adding a new service requires creating a new file, representing the service and implementing a few required methods.

To conclude

The tool has gone through major refactoring and got significantly improved since it was moved to its own repository in August 2016. If you’re a Tempest user, I’d recommend you try python-tempestconf if you haven’t already.

by mkopec at February 11, 2019 02:50 PM

Lars Kellogg-Stedman

In which I PEBKAC so you don't have to

Say you have a simple bit of code:

#include <avr/io.h>
#include <util/delay.h>

#define LED_BUILTIN _BV(PORTB5)

int main(void) 
{
    DDRB |= LED_BUILTIN;

    while (1)
    {
        PORTB |= LED_BUILTIN;   // turn on led
        _delay_ms(1000);        // delay 1s

        PORTB &= ~LED_BUILTIN;  // turn off led
        _delay_ms(1000);        // delay 1s
    }                                                
}

You have a Makefile that …

by Lars Kellogg-Stedman at February 11, 2019 05:00 AM

In which I PEBKAC so you don't have to

Say you have a simple bit of code: #include <avr/io.h> #include <util/delay.h> #define LED_BUILTIN _BV(PORTB5) int main(void) { DDRB |= LED_BUILTIN; while (1) { PORTB |= LED_BUILTIN; // turn on led _delay_ms(1000); // delay 1s PORTB &= ~LED_BUILTIN; // turn off led _delay_ms(1000); // delay 1s } } You have a Makefile that compiles that into an object (.o) file like this: avr-gcc -mmcu=atmega328p -DF_CPU=16000000 -Os -c blink.c If you were to forget to set the device type when compiling your .

February 11, 2019 12:00 AM

February 05, 2019

Carlos Camacho

TripleO - Deployment configurations

This post is a summary of the deployments I usually test for deploying TripleO using quickstart.

The following steps need to run in the Hypervisor node in order to deploy both the Undercloud and the Overcloud.

You need to execute them one after the other, the idea of this recipe is to have something just for copying/pasting.

Once the last step ends you can/should be able to connect to the Undercloud VM to start operating your Overcloud deployment.

The usual steps are:

01 - Prepare the hypervisor node.

Now, let’s install some dependencies. Same Hypervisor node, same root user.

# In this dev. env. /var is only 50GB, so I will create
# a sym link to another location with more capacity.
# It will take easily more tan 50GB deploying a 3+1 overcloud
sudo mkdir -p /home/libvirt/
sudo ln -sf /home/libvirt/ /var/lib/libvirt

# Disable IPv6 lookups
# sudo bash -c "cat >> /etc/sysctl.conf" << EOL
# net.ipv6.conf.all.disable_ipv6 = 1
# net.ipv6.conf.default.disable_ipv6 = 1
# EOL
# sudo sysctl -p

# Enable IPv6 in kernel cmdline
# sed -i s/ipv6.disable=1/ipv6.disable=0/ /etc/default/grub
# grub2-mkconfig -o /boot/grub2/grub.cfg
# reboot

sudo yum groupinstall "Virtualization Host" -y
sudo yum install git lvm2 lvm2-devel -y
sudo yum install libvirt-python python-lxml libvirt -y

02 - Create the toor user (from the Hypervisor node, as root).

sudo useradd toor
echo "toor:toor" | sudo chpasswd
echo "toor ALL=(root) NOPASSWD:ALL" \
  | sudo tee /etc/sudoers.d/toor
sudo chmod 0440 /etc/sudoers.d/toor
sudo su - toor

cd
mkdir .ssh
ssh-keygen -t rsa -N "" -f .ssh/id_rsa
cat .ssh/id_rsa.pub >> .ssh/authorized_keys
cat .ssh/id_rsa.pub | sudo tee -a /root/.ssh/authorized_keys
echo '127.0.0.1 127.0.0.2' | sudo tee -a /etc/hosts

export VIRTHOST=127.0.0.2
ssh root@$VIRTHOST uname -a

Now, follow as the toor user and prepare the Hypervisor node for the deployment.

03 - Clone repos and install deps.

git clone \
  https://github.com/openstack/tripleo-quickstart
chmod u+x ./tripleo-quickstart/quickstart.sh
bash ./tripleo-quickstart/quickstart.sh \
  --install-deps
sudo setenforce 0

Export some variables used in the deployment command.

04 - Export common variables.

export CONFIG=~/deploy-config.yaml
export VIRTHOST=127.0.0.2

Now we will create the configuration file used for the deployment, depending on the file you choose you will deploy different environments.

05 - Click on the environment description to expand the recipe.

OpenStack [Containerized & HA] - 1 Controller, 1 Compute

cat > $CONFIG << EOF
overcloud_nodes:
  - name: control_0
    flavor: control
    virtualbmc_port: 6230
  - name: compute_0
    flavor: compute
    virtualbmc_port: 6231
node_count: 2
containerized_overcloud: true
delete_docker_cache: true
enable_pacemaker: true
run_tempest: false
extra_args: >-
  --libvirt-type qemu
  --ntp-server pool.ntp.org
  -e /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml
EOF
OpenStack [Containerized & HA] - 3 Controllers, 1 Compute

cat > $CONFIG << EOF
overcloud_nodes:
  - name: control_0
    flavor: control
    virtualbmc_port: 6230
  - name: control_1
    flavor: control
    virtualbmc_port: 6231
  - name: control_2
    flavor: control
    virtualbmc_port: 6232
  - name: compute_1
    flavor: compute
    virtualbmc_port: 6233
node_count: 4
containerized_overcloud: true
delete_docker_cache: true
enable_pacemaker: true
run_tempest: false
extra_args: >-
  --libvirt-type qemu
  --ntp-server pool.ntp.org
  --control-scale 3
  --compute-scale 1
  -e /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml
EOF
OpenShift [Containerized] - 1 Controller, 1 Compute

cat > $CONFIG << EOF
# Original from https://github.com/openstack/tripleo-quickstart/blob/master/config/general_config/featureset033.yml
composable_scenario: scenario009-multinode.yaml
deployed_server: true

network_isolation: false
enable_pacemaker: false
overcloud_ipv6: false
containerized_undercloud: true
containerized_overcloud: true

# This enables TLS for the undercloud which will also make haproxy bind to the
# configured public-vip and admin-vip.
undercloud_generate_service_certificate: false
undercloud_enable_validations: false

# This enables the deployment of the overcloud with SSL.
ssl_overcloud: false

# Centos Virt-SIG repo for atomic package
add_repos:
  # NOTE(trown) The atomic package from centos-extras does not work for
  # us but its version is higher than the one from the virt-sig. Hence,
  # using priorities to ensure we get the virt-sig package.
  - type: package
    pkg_name: yum-plugin-priorities
  - type: generic
    reponame: quickstart-centos-paas
    filename: quickstart-centos-paas.repo
    baseurl: https://cbs.centos.org/repos/paas7-openshift-origin311-candidate/x86_64/os/
  - type: generic
    reponame: quickstart-centos-virt-container
    filename: quickstart-centos-virt-container.repo
    baseurl: https://cbs.centos.org/repos/virt7-container-common-candidate/x86_64/os/
    includepkgs:
      - atomic
    priority: 1

extra_args: ''

container_args: >-
  # If Pike or Queens
  #-e /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml
  # If Ocata, Pike, Queens or Rocky
  #-e /home/stack/containers-default-parameters.yaml
  # If >= Stein
  -e /home/stack/containers-prepare-parameter.yaml

  -e /usr/share/openstack-tripleo-heat-templates/openshift.yaml
# NOTE(mandre) use container images mirrored on the dockerhub to take advantage
# of the proxy setup by openstack infra
docker_openshift_etcd_namespace: docker.io/
docker_openshift_cluster_monitoring_namespace: docker.io/tripleomaster
docker_openshift_cluster_monitoring_image: coreos-cluster-monitoring-operator
docker_openshift_configmap_reload_namespace: docker.io/tripleomaster
docker_openshift_configmap_reload_image: coreos-configmap-reload
docker_openshift_prometheus_operator_namespace: docker.io/tripleomaster
docker_openshift_prometheus_operator_image: coreos-prometheus-operator
docker_openshift_prometheus_config_reload_namespace: docker.io/tripleomaster
docker_openshift_prometheus_config_reload_image: coreos-prometheus-config-reloader
docker_openshift_kube_rbac_proxy_namespace: docker.io/tripleomaster
docker_openshift_kube_rbac_proxy_image: coreos-kube-rbac-proxy
docker_openshift_kube_state_metrics_namespace: docker.io/tripleomaster
docker_openshift_kube_state_metrics_image: coreos-kube-state-metrics

deploy_steps_ansible_workflow: true
config_download_args: >-
  -e /home/stack/config-download.yaml
  --disable-validations
  --verbose
composable_roles: true

overcloud_roles:
  - name: Controller
    CountDefault: 1
    tags:
      - primary
      - controller
    networks:
      - External
      - InternalApi
      - Storage
      - StorageMgmt
      - Tenant
  - name: Compute
    CountDefault: 0
    tags:
      - compute
    networks:
      - External
      - InternalApi
      - Storage
      - StorageMgmt
      - Tenant

tempest_config: false
test_ping: false
run_tempest: false
EOF


From the Hypervisor, as the toor user run the deployment command to deploy both your Undercloud and Overcloud.

06 - Deploy TripleO.

bash ./tripleo-quickstart/quickstart.sh \
      --clean          \
      --release master \
      --teardown all   \
      --tags all       \
      -e @$CONFIG      \
      $VIRTHOST

Updated 2019/02/05: Initial version.

Updated 2019/02/05: TODO: Test the OpenShift deployment.

Updated 2019/02/06: Added some clarifications about where the commands should run.

by Carlos Camacho at February 05, 2019 12:00 AM

February 04, 2019

Matthias Runge

Collectd contributors meetup in Brussels

Last week, on January 31 and Feb 01, we had a contributors meetup of collectd in Brussels, just before FOSDEM. Thanks to our generous sponsors Camptocamp, Intel, and Red Hat, we were able to meet and to talk about various topics. In total, there were 15 persons attending, and if …

by mrunge at February 04, 2019 10:40 AM

February 01, 2019

Lars Kellogg-Stedman

ATOMIC_BLOCK magic in avr-libc

The AVR C library, avr-libc, provide an ATOMIC_BLOCK macro that you can use to wrap critical sections of your code to ensure that interrupts are disabled while the code executes. At high level, the ATOMIC_BLOCK macro (when called using ATOMIC_FORCEON) does something like this:

cli();

...your code here...

seti();

But …

by Lars Kellogg-Stedman at February 01, 2019 05:00 AM

ATOMIC_BLOCK magic in avr-libc

The AVR C library, avr-libc, provide an ATOMIC_BLOCK macro that you can use to wrap critical sections of your code to ensure that interrupts are disabled while the code executes. At high level, the ATOMIC_BLOCK macro (when called using ATOMIC_FORCEON) does something like this: cli(); ...your code here... seti(); But it’s more than that. If you read the documentation for the macro, it says: Creates a block of code that is guaranteed to be executed atomically.

February 01, 2019 12:00 AM

January 29, 2019

John Likes OpenStack

How do I re-run only ceph-ansible when using tripleo config-download?

After config-download runs the first time, you may do the following:


cd /var/lib/mistral/config-download/
bash ansible-playbook-command.sh --tags external_deploy_steps

The above runs only the external deploy steps, which for the ceph-ansible integration, means run the ansible which generates the inventory and then execute ceph-ansible.

More on this in TripleO config-download User’s Guide: Deploying with Ansible.

If you're using the standalone deployer, then config-download does not provide the ansible-playbook-command.sh. You can workaround this by doing the following:


cd /root/undercloud-ansible-su_6px97
ansible -i inventory.yaml -m ping all
ansible-playbook -i inventory.yaml -b deploy_steps_playbook.yaml --tags external_deploy_steps

The above makes the following assumptions:

  • You ran standalone with `--output-dir=$HOME` as root and that undercloud-ansible-su_6px97 was created by config download and contains the downloaded playbooks. Use `ls -ltr` to find the latest version.
  • If you're using the newer python3-only versions you ran something like `ln -s $(which ansible-3) /usr/local/bin/ansible`
  • That config-download already generated the overcloud inventory.yaml (the second command above is just to test that the inventory is working)

by Unknown (noreply@blogger.com) at January 29, 2019 10:26 PM

January 28, 2019

Lars Kellogg-Stedman

AVR micro-optimization: Losing malloc

Pssst! Hey...hey, buddy, wanna get an extra KB for cheap?

When write OO-style code in C, I usually start with something like the following, in which I use malloc() to allocate memory for a variable of a particular type, perform some initialization actions, and then return it to the …

by Lars Kellogg-Stedman at January 28, 2019 05:00 AM

AVR micro-optimization: Avr-gcc and --short-enums

How big is an enum?

I noticed something odd while browsing through the assembly output of some AVR C code I wrote recently. In the code, I have the following expression:

int main() {
    setup();

    while (state != STATE_QUIT) {
        loop();
    }
}

Here, state is a variable of type enum STATE, which looks something …

by Lars Kellogg-Stedman at January 28, 2019 05:00 AM

AVR micro-optimization: Losing malloc

Pssst! Hey…hey, buddy, wanna get an extra KB for cheap? When I write OO-style code in C, I usually start with something like the following, in which I use malloc() to allocate memory for a variable of a particular type, perform some initialization actions, and then return it to the caller: Button *button_new(uint8_t pin, uint8_t poll_freq) { Button *button = (Button *)malloc(sizeof(Button)); // do some initialization stuff return button; } And when initially writing pipower, that’s exactly what I did.

January 28, 2019 12:00 AM

AVR micro-optimization: Avr-gcc and --short-enums

How big is an enum? I noticed something odd while browsing through the assembly output of some AVR C code I wrote recently. In the code, I have the following expression: int main() { setup(); while (state != STATE_QUIT) { loop(); } } Here, state is a variable of type enum STATE, which looks something like this (not exactly like this; there are actually 19 possible values but I didn’t want to clutter this post with unnecessary code listings):

January 28, 2019 12:00 AM

January 22, 2019

Groningen Rain

Passing The Torch Without Dropping The Ball

And I was like, “That’s cool,” and my boss says it’s cool, everyone said it was cool, except that my uterus said that’s Not Cool because we’re going to have twins now.

by K Rain at January 22, 2019 09:00 AM

Lars Kellogg-Stedman

Debugging attiny85 code, part 3: Tracing with simavr

This is the third of three posts about using gdb and simavr to debug AVR code. The complete series is:

by Lars Kellogg-Stedman at January 22, 2019 07:00 AM

Debugging attiny85 code, part 2: Automating GDB with scripts

This is the second of three posts about using gdb and simavr to debug AVR code. The complete series is:

by Lars Kellogg-Stedman at January 22, 2019 06:00 AM

Debugging attiny85 code, part 1: simavr and gdb

In a case of awful timing, after my recent project involving some attiny85 programming I finally got around to learning how to use simavr and gdb to help debug my AVR code. It was too late for me (and I will never get the time back that I spent debugging …

by Lars Kellogg-Stedman at January 22, 2019 05:00 AM

Debugging attiny85 code, part 3: Tracing with simavr

This is the third of three posts about using gdb and simavr to debug AVR code. The complete series is: Part 1: Using GDB A walkthrough of using GDB to manually inspect the behavior of our code. Part 2: Automating GDB with scripts Creating GDB scripts to automatically test the behavior of our code. Part 3: Tracing with simavr Using simavr to collect information about the state of microcontroller pins while our code is running.

January 22, 2019 12:00 AM

Debugging attiny85 code, part 2: Automating GDB with scripts

This is the second of three posts about using gdb and simavr to debug AVR code. The complete series is: Part 1: Using GDB A walkthrough of using GDB to manually inspect the behavior of our code. Part 2: Automating GDB with scripts Creating GDB scripts to automatically test the behavior of our code. Part 3: Tracing with simavr Using simavr to collect information about the state of microcontroller pins while our code is running.

January 22, 2019 12:00 AM

Debugging attiny85 code, part 1: simavr and gdb

In a case of awful timing, after my recent project involving some attiny85 programming I finally got around to learning how to use simavr and gdb to help debug my AVR code. It was too late for me (and I will never get the time back that I spent debugging things with an LED and lots of re-flashing), but maybe this will help someone else! I’ve split this into three posts:

January 22, 2019 12:00 AM

January 19, 2019

OpenStack In Production (CERN)

OpenStack In Production - moving to a new home


During 2011 and 2012, CERN IT took a new approach to how to manage the infrastructure for analysing the data from the LHC and other experiments. The Agile Infrastructure project was formed covering service provisioning, configuration management and monitoring by adopting commonly used open source solutions with active communities to replace the in house tool suite.



In 2019, the CERN cloud managed infrastructure has grown by a factor of 10 compared to the resources in 2013. This has been achieved in collaboration with the many open source communities we have worked with over the past years, including
  • OpenStack
  • RDO
  • CentOS
  • Puppet
  • Foreman
  • Elastic
  • Grafana
  • Collectd
  • Kubernetes
  • Ceph
These experiences have been shared with over 40 blogs and more than 100 different talks at open source events during this time from small user groups to large international conferences. 

The OpenStack-In-Production blog has been covering the experiences, with a primary focus on the development of the CERN cloud service. However, the challenges of the open source world are now covering many more projects so it is time to move to a new blog to cover not only work on OpenStack but other communities and the approaches to integrated these projects into the CERN production infrastructure.

Thus, this blog will be moving to its new home at https://techblog.web.cern.ch/techblog/, incorporating our experiences with these other technologies. For those who would like to follow a specific subset of our activities, there are also taxonomy based content to select new OpenStack articles at https://techblog.web.cern.ch/techblog/tags/openstack/ and the legacy blog content at https://techblog.web.cern.ch/techblog/tags/openinfra/.

One of the most significant benefits we've found from sharing is receiving the comments from other community members. These often help to guide us in further investigations on solving difficult problems and to identify common needs to work on together upstream.

Look forward to hearing from you all on the techblog web site.

References



by Unknown (noreply@blogger.com) at January 19, 2019 09:31 AM

Lars Kellogg-Stedman

PiPower: A Raspberry Pi UPS

pipower top view

I have a Raspberry Pi running RetroPie hooked up to a television. It's powered from a USB port on the TV, which is convenient, but it means that whenever we shut off the TV we're pulling the plug on the Pi. While there haven't been any problems so far, this …

by Lars Kellogg-Stedman at January 19, 2019 05:00 AM

PiPower: A Raspberry Pi UPS

I have a Raspberry Pi running RetroPie hooked up to a television. It’s powered from a USB port on the TV, which is convenient, but it means that whenever we shut off the TV we’re pulling the plug on the Pi. While there haven’t been any problems so far, this is a classic recipe for filesystem problems or data loss at some point. I started looking into UPS options to alleviate this issue.

January 19, 2019 12:00 AM

January 07, 2019

John Likes OpenStack

ceph-ansible podman with vagrant

These are just my notes on how I got vagrant with libvirt working on CentOS7 and then used ceph-ansible's fedora29 podman tests to deploy a containerized ceph nautilus preview cluster without docker. I'm doing this in hopes of hooking Ceph into the new podman TripleO deploys.

Configure Vagrant with libvirt on CentOS7

I already have a CentOS7 machine I used for tripleo quickstart. I did the following to get vagrant working on it with libvirt.

1. Create a vagrant user


sudo useradd vagrant
sudo usermod -aG wheel vagrant
sudo usermod --append --groups libvirt vagrant
sudo su - vagrant
mkdir .ssh
chmod 700 .ssh/
cd .ssh/
curl https://github.com/fultonj.keys > authorized_keys
chmod 600 authorized_keys

Continue as the vagrant user.

2. Install the Vagrant and other RPMs

Download the CentOS Vagrant RPM from https://www.vagrantup.com/downloads.html and install other RPMs needed for it to work with libvirt.

sudo yum install vagrant_2.2.2_x86_64.rpm
sudo yum install qemu libvirt libvirt-devel ruby-devel gcc qemu-kvm
vagrant plugin install vagrant-libvirt

Note that I already had many of the libvirt deps above from quickstart.

3. Get a CentOS7 box for verification

Download the centos/7 box.

[vagrant@hamfast ~]$ vagrant box add centos/7
==> box: Loading metadata for box 'centos/7'
box: URL: https://vagrantcloud.com/centos/7
This box can work with multiple providers! The providers that it
can work with are listed below. Please review the list and choose
the provider you will be working with.

1) hyperv
2) libvirt
3) virtualbox
4) vmware_desktop

Enter your choice: 2
==> box: Adding box 'centos/7' (v1811.02) for provider: libvirt
box: Downloading: https://vagrantcloud.com/centos/boxes/7/versions/1811.02/providers/libvirt.box
box: Download redirected to host: cloud.centos.org
==> box: Successfully added box 'centos/7' (v1811.02) for 'libvirt'!
[vagrant@hamfast ~]$
Create a Vagrant file for it with `vagrant init centos/7`.

4. Configure Vagrant to use a custom storage pool (Optional)

Because I was already using libvirt directly with an images pool, vagrant was unable to download the centos/7 system. I like this as I want to keep my images pool separate for when I use libvirt directly. To make Vagrant happy I created my own pool for it and added the following to my Vagrantfile:


Vagrant.configure("2") do |config|
config.vm.provider :libvirt do |libvirt|
libvirt.storage_pool_name = "vagrant_images"
end
end

After doing the above `vagrant up` worked for me.

Run ceph-ansible's Fedora 29 podman tests

1. Clone ceph-ansible master

git clone git@github.com:ceph/ceph-ansible.git; cd ceph-ansible

2. Satisfy dependencies


sudo pip install -r requirements.txt
sudo pip install tox
cp vagrant_variables.yml.sample vagrant_variables.yml
cp site.yml.sample site.yml

Optionally: modify Vagrantfile for vagrant_images storage pool

3. Deploy with the container_podman


tox -e dev-container_podman -- --provider=libvirt

The above will result in tox triggering vagrant to create 10 virtual machines and then ceph-ansible will install ceph on them.

4. Inspect Deployment

Verify the virtual machines are running:


[vagrant@hamfast ~]$ cd ~/ceph-ansible/tests/functional/fedora/29/container-podman
[vagrant@hamfast container-podman]$ cp vagrant_variables.yml.sample vagrant_variables.yml
[vagrant@hamfast container-podman]$ vagrant status
Current machine states:

mgr0 running (libvirt)
client0 running (libvirt)
client1 running (libvirt)
rgw0 running (libvirt)
mds0 running (libvirt)
rbd-mirror0 running (libvirt)
iscsi-gw0 running (libvirt)
mon0 running (libvirt)
mon1 running (libvirt)
mon2 running (libvirt)
osd0 running (libvirt)
osd1 running (libvirt)

This environment represents multiple VMs. The VMs are all listed
above with their current state. For more information about a specific
VM, run `vagrant status NAME`.
[vagrant@hamfast container-podman]$

Connect to a monitor and see that it's running Ceph containers


[vagrant@hamfast container-podman]$ vagrant ssh mon0
Last login: Mon Jan 7 17:11:28 2019 from 192.168.121.1
[vagrant@mon0 ~]$

[vagrant@mon0 ~]$ sudo podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c494695eb0c2 docker.io/ceph/daemon:latest-master /opt/ceph-container... 4 hours ago Up 4 hours ago ceph-mgr-mon0
dbabf02df984 docker.io/ceph/daemon:latest-master /opt/ceph-container... 4 hours ago Up 4 hours ago ceph-mon-mon0
[vagrant@mon0 ~]$

[vagrant@mon0 ~]$ sudo podman images
REPOSITORY TAG IMAGE ID CREATED SIZE
docker.io/ceph/daemon latest-master 24fdc8c3cb3f 4 weeks ago 726MB
[vagrant@mon0 ~]$

[vagrant@mon0 ~]$ which docker
/usr/bin/which: no docker in (/home/vagrant/.local/bin:/home/vagrant/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin)
[vagrant@mon0 ~]$
Observe the status of the Ceph cluster:

[vagrant@mon0 ~]$ sudo podman exec dbabf02df984 ceph -s
cluster:
id: 9d2599f2-aec7-4c7c-a88e-7a8d39ebb557
health: HEALTH_WARN
application not enabled on 1 pool(s)

services:
mon: 3 daemons, quorum mon0,mon1,mon2 (age 71m)
mgr: mon1(active, since 70m), standbys: mon2, mon0
mds: cephfs-1/1/1 up {0=mds0=up:active}
osd: 4 osds: 4 up (since 68m), 4 in (since 68m)
rbd-mirror: 1 daemon active
rgw: 1 daemon active

data:
pools: 13 pools, 124 pgs
objects: 194 objects, 3.5 KiB
usage: 54 GiB used, 71 GiB / 125 GiB avail
pgs: 124 active+clean

[vagrant@mon0 ~]$

Observe the installed versions:


[vagrant@mon0 ~]$ sudo podman exec -ti dbabf02df984 /bin/bash
[root@mon0 /]# cat /etc/redhat-release
CentOS Linux release 7.6.1810 (Core)
[root@mon0 /]#

[root@mon0 /]# ceph --version
ceph version 14.0.1-1496-gaf96e16 (af96e16271b620ab87570b1190585fffc06daeac) nautilus (dev)
[root@mon0 /]#

Observe the OSDs


[vagrant@hamfast container-podman]$ vagrant ssh osd0
[vagrant@osd0 ~]$ sudo su -
[root@osd0 ~]# podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
4fe23502592c docker.io/ceph/daemon:latest-master /opt/ceph-container... About an hour ago Up About an hour ago ceph-osd-2
f582b4311076 docker.io/ceph/daemon:latest-master /opt/ceph-container... About an hour ago Up About an hour ago ceph-osd-0
[root@osd0 ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 50G 0 disk
sdb 8:16 0 50G 0 disk
├─test_group-data--lv1 253:1 0 25G 0 lvm
└─test_group-data--lv2 253:2 0 12.5G 0 lvm
sdc 8:32 0 50G 0 disk
├─sdc1 8:33 0 25G 0 part
└─sdc2 8:34 0 25G 0 part
└─journals-journal1 253:3 0 25G 0 lvm
vda 252:0 0 41G 0 disk
├─vda1 252:1 0 1G 0 part /boot
└─vda2 252:2 0 40G 0 part
└─atomicos-root 253:0 0 40G 0 lvm /sysroot
[root@osd0 ~]#

[root@osd0 ~]# podman exec 4fe23502592c cat var/lib/ceph/osd/ceph-2/type
bluestore
[root@osd0 ~]#

by Unknown (noreply@blogger.com) at January 07, 2019 08:07 PM

January 04, 2019

John Likes OpenStack

Simulate edge deployments using TripleO Standalone

My colleagues presented at OpenStack Summit Berlin on Distributed Hyperconvergence. This includes using TripleO to deploy a central controller node, extracting information from that central node, and then passing that information as input to a second TripleO deployment at a remote location ("on the edge of the network"). This edge deployment could host its own Ceph cluster which is collocated with compute nodes in its own availability zone. A third TripleO deployment could be added for a second remote edge deployment and users could then use the central deployment to schedule workloads per availability zone closer to where the workloads are needed.

You can simulate this type of deployment today with a single hypervisor and TripleO's standalone installer as per the newly merged upstream docs.

by Unknown (noreply@blogger.com) at January 04, 2019 07:38 PM

January 03, 2019

Adam Young

TripleO Networks from Simplest to Not-So-Simple

If you read the TripleO setup for network isolation, it lists eight distinct networks. Why does TripleO need so many networks? Lets take it from the ground up.

WiFi to the Workstation

WifI to the Workstation

I run Red Hat OpenStack Platform (OSP) Director, which is the productized version of TripleO.  Everything I say here should apply equally well to the upstream and downstream variants.

My setup has OSP Director running in a virtual machine (VM). To get that virtual machine set up takes network connectivity. I perform this via Wireless, as I move around the hose with my laptop, and the workstation has a built in wireless card.

Let’s start here: Director runs inside a virtual machine on the workstation.  It has complete access to the network interface card (NIC) via macvtap.  This NIC is attached to a Cisco Catalyst switch.  A wired cable to my laptop is also attached to the switch. This allows me to setup and test the first stage of network connectivity:  SSH access to the virtual machine running in the workstation.

Provisioning Network

The Blue Network here is the provisioning network.  This reflects two of the networks from the Tripleo document:

  • IPMI* (IPMI System controller, iLO, DRAC)
  • Provisioning* (Undercloud control plane for deployment and management)

These two distinct roles can be served by the same network in my setup, and, infact they must be.  Why?  Because my Dell servers have a  NIC that acts as both the IPMI endpoint and is the only NIC that supports PXE.  Thus, unless I wanted to do some serious VLAN wizardry, and get the NIC to switch both (tough to debug during the setup stage) I am better off with them both using untagged VLAN traffic.  Thus, each server is allocated two static IPv4 address, one to be used for IPMI, and one that will be assigned during the hardware provisioning.

Apologies for the acronym soup.  It bothers me, too.

Another way to think about the set of networks you need is via DHCP traffic.  Since the IPMI cards are statically assigned their IP addresses, they do not need a DHCP server.  But, the hardware’s Operating system will get its IP address from DHCP.  Thus, it is OK if these two functions share a Network.

This does not scale very well.  IPMI and IDrac can both support DHCP, and that would be the better way to go in the future, but is beyond the scope of what I am willing to mess with in my lab.

Deploying the Overcloud

In order to deploy the overcloud, the Director machine needs to perform two classes of network calls:

  1. SSH calls to the baremetal OS to lunch the services, almost all of which are containers.  This is on the Blue network above.
  2. HTTPS calls to the services running in those containers.  These services also need to be able to talk to each other.  This is on the Yellow internal API network above.  I didn’t color code “Yellow” as you can’t read it.  Yellow.

Internal (not) versus External

You might notice that my diagram has an additional network; the External API network is shown in Red.

Provisioning and calling services are two very different use cases.  The most common API call in OpenStack is POST https://identity/v3/auth/token.  This call is made prior to any other call.  The second most common is the call to validate a token.  The create token  call needs to be access able from everywhere that OpenStack is used.  The validate token call does not.  But, if the API server only  listens on the same network that is used for provisioning, that means the network is wide open;  people that should only be able to access the OpenStack APIs can now send network attacks against the IPMI cards.

To split this traffic, either the network APIs need to listen on both networks, or the provisioning needs to happen on the external API network. Either way, both networks are going to be set up when the overcloud is deployed.

Thus, the Red Server represents the API servers that are running on the controller, and the yellow server represents the internal agents that are running on the compute node.

Some Keystone History

When a user performs an action in the OpenStack system, they make an API call.  This request is processed by the webserver running on the appropriate controller host.  There is no difference between a Nova server requesting a token and project member requesting a token. These were seen as separate use cases, and were put on separate network ports.  The internal traffic was on port 35357, and the project member traffic was on port 5000.

It turns out that running on two different ports of the same IP address does not solve the real problem people were trying to solve.  They wanted to limit API access via network, not by port.  Thus, there really was no need for two different ports, but rather two different IP addresses.

This distinction still shows up in the Keystone service catalog, where Endpoints are classified as External or Internal.

Deploying and Using a Virtual Machine

Now Our Diagram has gotten a little more complicated.  Lets start with the newly added Red Lap top, attached to the External API network.  This system is used by our project member, and is used to create the new virtual machine via the compute create_server API call. In order:

  1. The API call comes from the outside world, travels over the Red external API network to the Nova server (shown in red)
  2. The Nova posts messages to the the Queue, which are eventually picked up and processed by the compute agent (shown in yellow).
  3. The compute agent talks back to the other API servers (also shown in Red) to fetch images, create network ports, and connect to storage volumes.
  4. The new VM (shown in green) is created and connects via an internal, non-routable IP address to the metadata server to fetch configuration data.
  5. The new VM is connected to the provider network (also shown in green).

At this point, the VM is up and running.  If an end user wants to connect to it they can do so.  Obviously, the Provider network does not run all the way through the router to the end users system, but this path is the “open for business” network pathway.

Note that this is an instance of a provider network as Assaf defined in his post.

Tenant Networks

Let say you are not using a provider network.  How does that change the setup?  First, lets re-label the Green network to be the “External Network.”  Notice that the virtual machines do not connect to it now.  Instead, they connect via the new, purple networks.

Note that the Purple networks connect to the external network in the network controller node, show in purple on the bottom server.  This service plays the role of a router, converting the internal traffic on the tenant network to the external traffic.  This is where the Floating IPs terminate, and are mapped to an address on the internal network.

Wrap Up

The TripleO network story has evolved to support a robust configuration that splits traffic into its component segments.  The diagrams above attempt to pass along my understanding of how they work, and why.

I’ve left off some of the story, as I do not show the separate networks that can be used for storage.  I’ve collapsed the controllers and agents into a simple block to avoid confusing detail, my goal is accuracy, but here it sacrifices precision.  It also only shows a simple rack configuration, much like the one here in my office.  The concepts presented should allow you to understand how it would scale up to a larger deployment.  I expect to talk about that in the future as well.

I’ll be sure to update  this article with feedback. Please let me know what I got wrong, and what I can state more clearly.

by Adam Young at January 03, 2019 05:27 PM

January 01, 2019

Giulio Fidente

Pushing Openstack and Ceph to the Edge at OpenStack summit Berlin 2018

We presented at the latest OpenStack (Open Infrastructure ?) summit, held in Berlin after te Rocky release, together with Sebastien Han and Sean Cohen, a session discussing how TripleO will support "Edge" deployments with storage at the edge; colocating Ceph with the OpenStack services in the edge zone yet keeping a small hardware footprint.

Thanks to the OpenStack Foundation for giving us the chance.

by Giulio Fidente at January 01, 2019 11:00 PM

December 14, 2018

Emilien Macchi

OpenStack Containerization with Podman – Part 4 (Healthchecks)

For this fourth episode, we’ll explain how we implemented healthchecks for Podman containers. Don’t miss the first, second and third episodes where we learnt how to deploy, operate and upgrade Podman containers.

In this post, we’ll see the work that we have done to implement container healthchecks with Podman.

Note: Jill Rouleau wrote the code in TripleO to make that happen.

Context

Docker can perform health checks directly in the Docker engine without the need of an external monitoring tool or sidecar containers.

A script (usually per-image) would be run by the engine and the return code would define if whether or not a container is healthy.

Example of healthcheck script:

curl -g -k -q --fail --max-time 10 --user-agent curl-healthcheck \
--write-out "\n%{http_code} %{remote_ip}:%{remote_port} %{time_total} seconds\n" https://my-app:8774 || return 1

It was originally built so unhealthy containers can be rescheduled or removed by Docker engine. The health could be verified by docker ps or docker inspect commands:

$ docker ps
my-app "/entrypoint.sh" 30 seconds ago Up 29 seconds (healthy) 8774/tcp my-app

However with Podman we don’t have that kind of engine anymore but having that monitoring interface has been useful in our architecture, so our operators can use this interface to verify the state of the containers.

Several options were available to us:

  • systemd timers (like cron) to schedule the health checks. Example documented on coreos manuals.
  • use Podman Pods with Side Car container running health checks.
  • add a scheduling function to conmon with a scheduling function.
  • systemd service, like a podman-healthcheck service that would run on a fixed interval.

If you remember from the previous posts, we decided to get some help from systemd to control the containers, like automatically restart on failure and also automatic start at boot. With that said, we decided to go with the first option, which seems the easier to integrate and the less invasive.

Implementation

The systemd timer is a well-known mechanism. It’s basically a native feature in systemd that allows to run a specific service in a time controlled. The service would be a “OneShot” type, executing the healthcheck script present in the container image.

Here is how we did for our OpenStack containers (with a timer of 30 seconds for healthchecks, configurable like a cron):

# my_app_healthcheck.timer

[Unit]
Description=my_app container healthcheck
Requires=my_app_healthcheck.service
[Timer]
OnUnitActiveSec=90
OnCalendar=*-*-* *:*:00/30
[Install]
WantedBy=timers.target
# my_app_healthcheck.service

[Unit]
Description=my_app healthcheck
Requisite=my_app.service
[Service]
Type=oneshot
ExecStart=/usr/bin/podman exec my_app /bin/healthcheck
[Install]
WantedBy=multi-user.target

Activate the timer and service:

$ systemctl daemon-reload
$ systemctl enable --now my_app_healthcheck.service
$ systemctl enable --now my_app_healthcheck.timer

Check the service & timer status:

$ service my_app_healthcheck status
Redirecting to /bin/systemctl status my_app_healthcheck.service
● my_app_healthcheck.service - my_app healthcheck
   Loaded: loaded (/etc/systemd/system/my_app_healthcheck.service; enabled; vendor preset: disabled)
   Active: activating (start) since Fri 2018-12-14 20:11:00 UTC; 158ms ago
 Main PID: 325504 (podman)
   CGroup: /system.slice/my_app_healthcheck.service
           └─325504 /usr/bin/podman exec my_app /bin/healthcheck
Dec 14 20:11:00 myhost.localdomain systemd[1]: Starting my_app healthcheck...

$ service my_app_healthcheck.timer status
Redirecting to /bin/systemctl status my_app_healthcheck.timer
● my_app_healthcheck.timer - my_app container healthcheck
   Loaded: loaded (/etc/systemd/system/my_app_healthcheck.timer; enabled; vendor preset: disabled)
   Active: active (waiting) since Fri 2018-12-14 18:42:22 UTC; 1h 30min ago
Dec 14 18:42:22 myhost.localdomain systemd[1]: Started my_app container healthcheck.

$ systemctl list-timers
NEXT                         LEFT          LAST                         PASSED       UNIT                                               ACTIVATES
Fri 2018-12-14 20:14:00 UTC  361ms left    Fri 2018-12-14 20:13:30 UTC  29s ago      my_app_healthcheck.timer               my_app_healthcheck.service

Now it’s implemented, let’s try it!

Demo

Stay in touch for the next post in the series of deploying TripleO and Podman!

Source of the demo.

by Emilien at December 14, 2018 07:26 PM

December 13, 2018

Groningen Rain

Who Down With OCP?!?

YEAH YOU KNOW ME! I went to six conferences in forty eight days and that’s just a bit overwhelming. I had this abstract idea that it was over the course of the around eight weeks or so but then I looked at the actual dates and pulled it up on a calendar calculator and now …

by K Rain at December 13, 2018 09:00 AM

December 05, 2018

RDO Blog

Community Blog Round Up 05 December 2018

Adam Young discusses OpenStack’s access policy, then deep dives to create a self trust in Keystone while Lars Kellogg-Stedman helps us manage USB gadgets using systemd as well as using ansible to integrate a password management service, then Pablo Iranzo Gómez shows how OpenStack contributions are peer reviewed.

Scoped and Unscoped access policy in OpenStack by Adam Young

Ozz did a fantastic job laying out the rules around policy. This article assumes you’ve read that. I’ll wait. I’d like to dig a little deeper into how policy rules should be laid out, and a bit about the realities of how OpenStack policy has evolved. OpenStack uses the policy mechanisms describe to limit access to various APIs. In order to make sensible decisions, the policy engine needs to know some information about the request, and the user that is making it.

Read more at https://adam.younglogic.com/2018/11/scoped-and-unscoped-access-policy-in-openstack/

Systemd unit for managing USB gadgets by Lars Kellogg-Stedman

The Pi Zero (and Zero W) have support for acting as a USB gadget: that means that they can be configured to act as a USB device — like a serial port, an ethernet interface, a mass storage device, etc. There are two different ways of configuring this support. The first only allows you to configure a single type of gadget at a time, and boils down to: Enable the dwc2 overlay in /boot/config.txt, Reboot, modprobe g_serial.

Read more at https://blog.oddbit.com/2018/10/19/systemd-unit-for-managing-usb-/

Integrating Bitwarden with Ansible by Lars Kellogg-Stedman

Bitwarden is a password management service (like LastPass or 1Password). It’s unique in that it is built entirely on open source software. In addition to the the web UI and mobile apps that you would expect, Bitwarden also provides a command-line tool for interacting with the your password store.

Read more at https://blog.oddbit.com/2018/10/19/integrating-bitwarden-with-ans/

Creating a Self Trust In Keystone by Adam Young

Lets say you are an administrator of an OpenStack cloud. This means you are pretty much all powerful in the deployment. Now, you need to perform some operation, but you don’t want to give it full admin privileges? Why? well, do you work as root on your Linux box? I hope note. Here’s how to set up a self trust for a reduced set of roles on your token.

Read more at https://adam.younglogic.com/2018/10/creating-a-self-trust-in-keystone/

Contributing to OSP upstream a.k.a. Peer Review by Pablo Iranzo Gómez

In the article “Contributing to OpenStack” we did cover on how to prepare accounts and prepare your changes for submission upstream (and even how to find low hanging fruits to start contributing). Here, we’ll cover what happens behind the scene to get change published.

Read more at https://iranzo.github.io/blog/2018/10/16/osp-upstream-contribution/

by Rain Leander at December 05, 2018 04:25 PM

November 26, 2018

Arie Bregman

OpenStack: Testing Upgrades with Tobiko

What is Tobiko?

Tobiko is an OpenStack upgrade testing framework. It aims to provide tooling and methods for easily developing upgrade type tests.

Note: At this moment it also provides you with several built-in networking tests.

If you are familiar with OpenStack you might wonder why the current OpenStack testing framework (Tempest) is not used for that purpose.

First of all, that’s an excellent question. Secondly, imagine the following scenario: you would like to set up several different OpenStack resources (instances, routers, networks, …) before upgrading the cloud, run the upgrade process and once the upgrade process is finished, run the same or similar tests to check if the resources you created are still there and working properly. Another scenario is to test your cloud during the upgrade and analyze what happens to the different resources while upgrading your cloud.

Tempest designed to run, test and cleanup a resource once the test is over. If you would try to run the same test before and after an update it would just create and remove the resources in each phase (pre and post) instead of cleaning them only post-upgrade.

Now, let’s move to the practical part of this post and learn how to use it.

How to install Tobiko?

Run the following code in order to install Tobiko in its own (Python) virtual environment

git clone https://review.openstack.org/openstack/tobiko.git
cd tobiko && pipenv install .

Verify it works by running a simple command like tobiko-list templates. It should list the Heat templates available in Tobiko

[abregman]$ tobiko-list --templates

test_floatingip.yaml
test_mtu.yaml

Running tests

Tobiko at this point is coupled to Tempest. It acts as tempest plugin so you execute it the following way

tempest run --config-file etc/tempest.conf --regex tobiko\.\*test_pre

This will run all the pre-upgrade tests. In order to run the post tests, you simply change the regex

tempest run --config-file etc/tempest.conf --regex tobiko\.\*test_post

To run successfully, it will require you to provide a valid tempest configuration file which provides it with input it uses like authentication method and image to use.

For some commands, Tobiko merely requires the authentication info which can be provided with environment variables and doesn’t require you to pass a whole tempest configuration file.

How it works?

Each test class in Tobiko is associated with a stack/template. When a test runs, it uses the template (which is located with other templates in tobko/tests/scenario/templates)  to create the stack.

Both stack and template are named the same as the test file. This means that if your tests file called “test_floatingip.py” then the stack name is “test_floatingip” and template name is “test_floatingip.yaml”

If the stack already exists, Tobiko will simply skip the stack creation and proceed to run the tests.

Remember, once the test is over it will not remove the resources/stack. There is a util allows you to remove resources created by Tobiko called ‘tobiko-delete. We’ll cover it later on.

Adding a test

First, add a setUp method that will call the base class (that used by each test class) for creating the stack required by the tests. In order to create the stack, the base class needs the file name (of the test class)

def setUp(self):
        super(<YOUR_TEST_CLASS>, self).setUp(__file__)

Next, make sure you have a test for each phase – pre and post. It should be called test_pre_… and test_post_… to match other existing tests included in Tobiko.

Tobiko CLI

Tobiko provides you with tooling in order to ease the process of developing new tests and testing related resources.

tobiko-list

tobiko-list lists the templates defined in Tobiko

tobiko-list --templates

test_floatingip.yaml
test_security_groups.yaml

It can also be used to list the stacks in your OpenStack project but only those that related to Tobiko (either created by it or match a template name defined in the project)

tobiko-list --stacks

test_floatingip

tobiko-create

If you would like only to create the stacks defined in Tobiko and not run the tests, you can use tobiko-create for that purpose.

You can either specify a specific stack the following way

tobiko-create -s test_floatingip

Or create all stacks

tobiko-create --all

tobiko-delete

Once you finished testing/developing, you can use tobiko-delete to clean up resources created during your development/testing. Similarly to tobiko-create you can delete a specific task

tobiko-delete -s test_floatingip

Or all stacks

tobiko-delete --all

Don’t worry about deleting all the stacks in your project. It will only delete stacks created by Tobiko or those that match templates names.

Contributions

Tobiko code review is managed via OpenStack Gerrit system. Follow OpenStack Developer guide to submit patches. The short version is: clone the project, commit your changes and run git-review to submit your patch for a review. The long version is well documented by OpenStack.

by Arie Bregman at November 26, 2018 02:59 PM

November 22, 2018

Adam Young

Scoped and Unscoped access policy in OpenStack

Ozz did a fantastic job laying out the rules around policy. This article assumes you’ve read that. I’ll wait.

I’d like to dig a little deeper into how policy rules should be laid out, and a bit about the realities of how OpenStack policy has evolved.

OpenStack uses the policy mechanisms describe to limit access to various APIs. In order to make sensible decisions, the policy engine needs to know some information about the request, and the user that is making it.

There are two classes of APIs in OpenStack, scoped and unscoped. A Scoped API is one where the resource is assigned to a project, or possibly, a domain. Since Domains are only used for Keystone, we’ll focus on projects for now. The other class of APIs are where the resources are not scoped to a project or domain, but rather belong to the cloud as a whole. A good example is a Nova Hypervisor.

The general approach to accessing scoped resources is to pass two checks. The first check is that the auth-data associated with the token has one of the appropriate roles in it. The second is that the auth-data is scoped to the same project as the resource.

For an example, lets look at the cinder API to for volumes. The API to create a new volume is:

POST /v3/{project_id}/volumes

and the API to then read the volume is

GET /v3/{project_id}/volumes/{volume_id}

The default policy.yaml for these APIs shows as:

# Create volume.
# POST /volumes
#"volume:create": ""

# Show volume.
# GET /volumes/{volume_id}
#"volume:get": "rule:admin_or_owner"

We’ll dig a little deeper into these in a moment.

One thing that distinguishes Cinder from many other APIs is that it
includes the project ID in the URL. This makes it easier to see what
the policy is that we need to enforce. For example, if I have a
Project ID of a226dc9813f745e19ece3d60ac5a351c and I want to create a
volume in it, I call:

POST https://cinderhost/v3/a226dc9813f745e19ece3d60ac5a351c/volumes

With the appropriate payload. Since the volume does not exist yet, we
have enough information to enforce policy right up front. If the
token I present has the following data in it:

{
  "token": {
    "methods": [
       "password"
     ],
     "roles": [
        {
            "id": "f03fda8f8a3249b2a70fb1f176a7b631",
             "name": "Member"
        }
     ],
     "project": {
        "id": "a226dc9813f745e19ece3d60ac5a351c",
        "domain": {
             "id": "default",
             "name": "Default"
        },
        "enabled": true,
        "description": null,
        "name": "tenant_name1"
     },
  }
}

Lets take another look at the policy rule to create a volume:

"volume:create": ""

There are no restrictions placed on this API. Does this mean that
anyone can create a volume? Not quite.

Just because oslo-policy CAN be used to enforce access does not mean
it is the only thing that does so. Since each of the services in
OpenStack have had long lives of their own, we find quirks like this.
In this case, the URL structure that has the project ID in it is
checked against the token externally to the oslo-policy check.

It also means that no role is enforced on create. Any user, with any
role on the project can create a volume.

What about afterwards? The rule on the get command is

"volume:get": "rule:admin_or_owner"

Here’s another gotcha. Each service has its own definition of what is
meant by an owner. You need to look at the service specific definition
of the rule to see what this means.

# Default rule for most non-Admin APIs.
#"admin_or_owner": "is_admin:True or (role:admin and is_admin_project:True) or project_id:%(project_id)s"

If you have understood the earlier two articles, you should be able to
interpret most of this rule. Lets start with the rightmost section:

`or project_id:%(project_id)s"`

The or rule means that, even if everything before this failed, we can
still pass if we pass only the part that follows. In this case, it is
doing the kind of scope check I described above: that the project_id
on from the token’s auth-data matches the project_id on the volume
object. While this is Cinder, and it is still doing the check based
on the URL, it aslo checks based on the resource, in this case the
volume. That means that this chech can’t happen until Cinder fetches
the Volume record from the database. There is no role check on this
API. A user with any role assigned on the project will be able to
execute the API.

What about the earlier parts of the rule? Lets start with the part we
can explain with the knowledge we have so far:

`role:admin`

This is a generic check that the user has the role assigned on the
token. If we were to look at this rule a couple years ago, this would have
been the end of the check. Instead, we see it is coupled with

`and is_admin_project:True`

This is an additional flag on the token’s auth data. It is attempting
to mitigate one of the oldest bugs in the bug tracker.

Bug 968696: “admin”-ness not properly scoped

Another way to describe this bug is to say that most policy rules were
written too permissive. A user that was assigned the `admin` role
anywhere ended up having `admin` permissions everywhere.

This breaks the scoping concept we discussed earlier.

So, what this flag implies is that the project that the user’s token is scoped
to is designated as the `admin` project in Keystone. If this is the
case, the token will have this additional flag set.

Essentially, the `admin` project is a magic project with elevated
privileges.

This provides a way to do cloud-wide administration tasks.

What about that first rule:

`is_admin:True`

This is a value set by the code inside the Cinder service. A similar
pattern exists in most projects in OpenStack. It is a way for cinder
to be able to override the policy check for internal operations. Look
in the code for places that call get_admin_context() such as:

volume_types = db.volume_type_get_all(context.get_admin_context(),
False)

What about those unscope APIs we were looking at earlier? It turns
out, they are mostly implemented with the first half of the cinder
Rule. For example, Update cluster has the policy rule

# PUT /clusters/{cluster_id}
"clusters:update": "rule:admin_api"

which is implemented as

# Default rule for most Admin APIs.
"admin_api": "is_admin:True or (role:admin and is_admin_project:True)"

One requirement that the Operator community had was that they needed to be able to do cloud wide operations, even when the operations should have been scoped to a project. List all VMs, list all users, and other types of operations were allowed to happen with admin-scoped tokens. This really obscured the difference between globlal and project scoped operations.

The is_admin_project hack works, but it is a bit esoteric. One current effort in the Keystone community is to do something a little more readable: actully have peroper scoping for things that are outside of projects. We are calling this Service scoping. Service scoped roles are available in the Rocky release, and can be used much as is_admin_project to mitigate bug 968696.

by Adam Young at November 22, 2018 02:56 PM

November 09, 2018

RDO Blog

Changing Default Blog Post Style to Left Instead of Justified

Bacon ipsum dolor amet ground round cow kevin tail buffalo tongue. Bacon biltong kevin, beef ribs shoulder capicola spare ribs meatloaf swine jerky sirloin. Beef ribs tenderloin porchetta short loin tri-tip pancetta pork chop pork strip steak pork loin flank corned beef turkey spare ribs meatball. Landjaeger picanha filet mignon, beef kielbasa spare ribs cow swine. Bacon ipsum dolor amet ground round cow kevin tail buffalo tongue. Bacon biltong kevin, beef ribs shoulder capicola spare ribs meatloaf swine jerky sirloin. Beef ribs tenderloin porchetta short loin tri-tip pancetta pork chop pork strip steak pork loin flank corned beef turkey spare ribs meatball. Landjaeger picanha filet mignon, beef kielbasa spare ribs cow swine.

Brisket jerky beef ribs swine. Tenderloin pig cupim ham hock, short loin bresaola biltong pork loin shank sirloin andouille turkey. Sirloin ribeye biltong t-bone venison. Chuck tail strip steak, swine kevin meatball andouille leberkas frankfurter capicola cupim cow. Cow turkey capicola, pastrami short ribs fatback kevin. Shoulder shankle flank short ribs t-bone. Sirloin tongue biltong, shoulder boudin landjaeger salami leberkas picanha flank swine tenderloin ground round t-bone prosciutto. Tenderloin t-bone jerky jowl, swine beef kevin ribeye doner pig biltong leberkas chicken. Kevin t-bone frankfurter chicken, short loin pork ball tip meatloaf shank meatball swine short ribs ham hock.

Fatback shoulder chuck landjaeger doner shank pork loin biltong sirloin beef short loin cow kielbasa swine. Hamburger bresaola porchetta short loin, leberkas cupim buffalo prosciutto meatloaf filet mignon ham turkey t-bone andouille. Leberkas meatloaf filet mignon rump. Venison t-bone leberkas, landjaeger beef ribs shank drumstick meatloaf kevin burgdoggen. Brisket jerky beef ribs swine. Tenderloin pig cupim ham hock, short loin bresaola biltong pork loin shank sirloin andouille turkey. Sirloin ribeye biltong t-bone venison. Chuck tail strip steak, swine kevin meatball andouille leberkas frankfurter capicola cupim cow. Cow turkey capicola, pastrami short ribs fatback kevin. Shoulder shankle flank short ribs t-bone. Sirloin tongue biltong, shoulder boudin landjaeger salami leberkas picanha flank swine tenderloin ground round t-bone prosciutto. Tenderloin t-bone jerky jowl, swine beef kevin ribeye doner pig biltong leberkas chicken. Kevin t-bone frankfurter chicken, short loin pork ball tip meatloaf shank meatball swine short ribs ham hock.

Fatback shoulder chuck landjaeger doner shank pork loin biltong sirloin beef short loin cow kielbasa swine. Hamburger bresaola porchetta short loin, leberkas cupim buffalo prosciutto meatloaf filet mignon ham turkey t-bone andouille. Leberkas meatloaf filet mignon rump. Venison t-bone leberkas, landjaeger beef ribs shank drumstick meatloaf kevin burgdoggen.

by Rain Leander at November 09, 2018 01:54 PM

November 05, 2018

Arie Bregman

OpenStack: Heat Python Tutorial

In this tutorial, we’ll focus on how to interact with OpenStack Heat using Python.  Before deep diving into Heat Python examples, I suggest being familiar with Heat itself and more specifically:

  • Templates
  • Basic operations: create/delete/update stack

Still here? let’s go 🙂

Set up Heat client

In order to work with Heat, we need first to create a heat client.

from heatclient import client as heat_client
from keystoneauth1 import loading
from keystoneauth1 import session

kwargs = {
'auth_url': ,
'username':,
'password': ,
'project_name': ,
'user_domain_name': ,
'project_domain_name': 
}

loader = loading.get_plugin_loader('password')
auth = loader.load_from_options(**kwargs)
sess = session.Session(auth=auth, verify=False)

client = heat_client.Client('1', session=sess, endpoint_type='public', service_type='orchestration')

Note: if for some reason you are using auth v2 and not v3, you can drop user_domain_name and project_domain_name.

You should be able to use your heat client now. Let’s test it.

List Stacks

for stack in client.stacks.list():
    print(stack)

Stack 
u'description':u'',
u'parent':None,
u'deletion_time':None,
u'stack_name':u'default',
u'stack_user_project_id':u'48babe632349f9b87ac3513',
u'stack_status_reason':u'Stack CREATE completed successfully',
u'creation_time':   u'2018-10-25T17:02:52   Z',
u'links': 
 
u'href':         u'https://my-server',
u'rel':u'self'
}
],
u'updated_time':None,
u'stack_owner':None,
u'stack_status':u'CREATE_COMPLETE',
u'id':u'b90d0e57-05a8-4700-b2f9-905497abe673',
u'tags':None
}>

The list method provides us with a generator that returns Stack objects. Each Stack object contains plenty of information. Information like the name of the stack, if it’s a nested stack you’ll get details on the parent stack, creation time and probably the most useful one – stack status which allows us to check if the stack is ready to use.

Create a Stack

In order to create a stack, we first need a template that will define how our stack would look like. I’m going to assume here that you read the template guide and you have a basic (or complex) template ready for use.

To load a template, heat developers have provided us with the get_template_content method

from heatclient.common import template_utils
import yaml

template_path = '/home/mario/my_template'

# Load the template
_files, template = template_utils.get_template_contents(template_path)

# Searlize it into a stream
s_template = yaml.safe_dump(template)

client.stacks.create(stack_name='my_stack', template = s_template)

Stack with parameters

In reality, there is a good chance your template includes several parameters that you have to pass when creating the stack. For example, take a look at this template

heat_template_version: 2013-05-23
description: My Awesome Stack

parameters:
  flavor:
    type: string
  image:
    type: string

In order for the stack creation to be completed successfully, we need to provide the parameters flavor and image. This will require a slight change in our code

parameters = {'flavor': 'm1.large', 'image': 'Fedora-30'}

client.stacks.create(stack_name='my_stack', template = s_template, parameters=parameters)

We created a dictionary with the required parameters and passed it to the stack create method. When more parameters added to your template, all you need to do is to extend the ‘parameters’ dictionary, without modifying the create call.

Inspect stack resources

Inspecting the stack as we previously did, might not be enough in certain scenarios. Imagine you want to use some resources as soon as they ready, regardless of overall stack readiness. In that case, you’ll want to check what is the status of a single resource. The following code will allow you to achieve that

stack = client.stacks.get("my_stack")
res = client.resources.get(stack.id, 'fip')
if res.resource_status == 'CREATE_COMPLETE':
    print("You may proceed :)")

So what did just happened? first, we need to obtain the ID of our stack. In order to do that we use the stacks get method by passing our stack’s name.

Now that we have the stack ID we can use it and the resource name we are interested in (‘fip’) to get the resource object.

Once we get the resource object, we can use ‘resource_status’ to check the if the stack creation has been completed and proceed accordingly.

Stack outputs

A better way to get quickly the output we interested in is the outputs section in Heat templates.

outputs:
server_ip:
  value: {get_attr: [floating_ip, floating_ip_address]}
server_ip2:
  value: {get_attr: [floating_ip2, floating_ip_address]}

In the above example, we are providing the user the information about two floating IPs of two different servers. We can then access this information with Python this way

print(my_stack.outputs)

[{u'output_value': u'10.2.224.22', u'output_key': u'server_ip2', u'description': u'No description given'}, {u'output_value': u'10.2.230.22', u'output_key': u'server_ip', u'description': u'No description given'}]

IP = stack.output[0]['output_value']
print(IP)

10.2.224.22

As you can see, each output represented by its own item in the ‘outputs’ list and is a better method (in my opinion at least) to access information quickly than inspecting the resources.

by Arie Bregman at November 05, 2018 08:41 PM

October 19, 2018

Lars Kellogg-Stedman

Systemd unit for managing USB gadgets

The Pi Zero (and Zero W) have support for acting as a USB gadget: that means that they can be configured to act as a USB device -- like a serial port, an ethernet interface, a mass storage device, etc.

There are two different ways of configuring this support. The first …

by Lars Kellogg-Stedman at October 19, 2018 04:00 AM