I wanna be a Craftsman. Thoughts about software engineering. by Roy Tsabari

[example] E2E Tests by Docker-Compose

docker
docker-compose

The App

A classic web aplication that allow CRUD operations on business data by http requests. The application is using MySql to store the data and post to ElasticSearch for easy search.

The Tests

We want to run a real end-to-end tests. We’ll load the service with a local Mysql and ES and send some http requests. Then we’lll check the data in the DB and ES to verify the changes has been made.

Show Me Some Code!

# docker-sompose.yaml

# docs: https://docs.docker.com/compose/compose-file/
version: '3'
services:
  my_app:
    container_name: my_app
    build: ./
    image: ${FULL_CONTAINER_NAME}
    environment:
      - ENV=test
      - ES_HOST=elasticsearch
      - DB_HOST=mysql
      - DB_NAME=db
      - DB_USER=root
      - DB_PASS=abc123
    ports:
      - 8080:8080
    depends_on:
      - mysql
      - elasticsearch
    # source: https://github.com/eficode/wait-for
    # `depends_on` is not enough.
    # We use wait-for to start the service only after Mysql and ES init is done
    # consider moving to docker's healthcheck
    command: ./wait-for elasticsearch:9200 --timeout=90 -- ./wait-for mysql:3306 --timeout=90 -- make start
    links:
      - mysql
      - elasticsearch
    logging:
      driver: json-file

  my_app_e2etests:
    container_name: my_app_e2etests
    build: ./
    image: ${FULL_CONTAINER_NAME}
    environment:
      - ENV=develop
      # We make some changes in ES and Mysql in tests init and read from it in the assertion phase
      - ES_HOST=elasticsearch
      - DB_HOST=mysql
      - DB_NAME=db
      - DB_USER=root
      - DB_PASS=abc123
    depends_on:
      - my_app
      - elasticsearch
      - mysql
    command: ./wait-for claims:8080 --timeout=60 -- npm run ci-test
    links:
      - elasticsearch
      - mysql
      - my_app
    volumes:
      # We save the test reports in JunitXML format so the CI can show them in a nice way
      - ${TESTS_OUTPUT_PATH}:/app/test-output/
    logging:
      driver: json-file

  mysql:
    container_name: mysql
    image: mysql:5.6
    environment:
      - MYSQL_ROOT_PASSWORD=abc123
      - MYSQL_DATABASE=db
    volumes:
    # The content of /e2e_tests/init_db/:
    # 1.create-tables.sql
    # 2.insert-data.sql
    #
    # These sql statements files will run automatically after DB init is done
      - ./e2e_tests/init_db/:/docker-entrypoint-initdb.d
    logging:
      driver: json-file

  elasticsearch:
    container_name: elasticsearch
    image: elasticsearch:5.6
    ports:
      - 9200:9200
    # https://docs.docker.com/compose/compose-file/#ulimits
    # https://www.ibm.com/support/knowledgecenter/en/ssw_aix_71/com.ibm.aix.cmds5/ulimit.htm
    # https://docs.docker.com/compose/compose-file/#sysctls
    ulimits:
      nofile: 65536
      #threads:
        #soft: 4096
        #hard: 4096
    logging:
      driver: json-file

How to run

# MakeFile
start:
	# Start the service
e2e-test:
	# Run the actual tests using Mocha, Pytest or anything else
docker-e2e-test:
	docker-compose rm -f mysql
	docker-compose rm -f  elasticsearch  
	# consider moving to --exit-code-from=my_app_e2etests  
	docker-compose up -t 300 --abort-on-container-exit

Access from container to host

docker
osx

docker run has a networking configuration you can specify. If you pass: --network=host the container will use the host network stack.

It doesn’t work on OSX. Whell, it actually does work, but you can’t really use it. Docker deamon on osx runs on a virtual machine, so passing --network=host use the VM network.

The simplest workaround I found is released in version 17.06 of docker-for-mac. Just use docker.for.mac.localhost as the host name.

links:

Your API should cover all the possible ways to use it

A good API guide your clients toward the right way to use it. Anyway, you should think about all the ways clients can use it, and handle these cases. Here are some examples:

Example 1 - config calss (Java):

class Config {
 public init () {
    // load config from file
 }
 
 public get (String key) {
    // return config value
 }
}

You want the users to call init() and then get(), but what if they call get() without init()? Sure, the best to way would be to force the client into the correct way to use it, and load the config from file in ctor. But let’s say you don’t want to use the ctor. Example: you are using a dependency injection framework that create the class instance for you and you want you client to be able to control when the havy loading happens.

You got two options here:

  1. Throw with a proper message if init is not called
  2. Init the configuration data is not initialized before.

Anyway, don’t leave this case unhandled.

Example 2 - convert module (Javascript):

You got a module that gets an object and convert some of its properties format.

function convert (value){
  //do some conversion and return
}
module.exports = data => {
  data.someProp = convert(data.someOtherProp);
}

With this API, your client needs to know that you are changing the parameter you got (which is debatable by itself). A better way would be to return the changed object, so event clients that use the returned value will get a good result:

A side note: I had a similar case, where I had many calculated fields, and I wanted to force a specific pattern so users can’t misuse the code. Javascript does not offer something a way to prevent function from change its parameter, but I found a nice solution: I added a .jshint file to the calculated fields folder - where all the calc. fields modules are, with one rule:

{
  "rules":{
    "no-param-reassign": ["error", { "props": true }]
  }
}

Basically it says that you can’t modify the function parameter. Eslint supports hierarchy in configuration, so all the other rules from the project .eslintrc file apply in this folder in addition to this specific rule.

Can you solve this puzzle?

Update - the position is not open anymore. you can still answer the puzzle though…


Outbrain is looking for experienced, talented, enthusiastic web developers. You can read about the position here. Interested? Solve this quick puzzle and contact me: roy@royts.com

// We need to create a chat client. 
// The messages pulling policy to should be:

// 1. Get the latest messages from  the server
// 2. Wait 2 sec.
// 3. Back to no. 1 again

// Q: what is wrong with the following solution? 
//    What would be the right way to do that?

setInterval(function () {
 
    $.getJSON( "http://chat-server.com/10/message", function( data ) {
          updateChatWindow(data.messages);
      });
 
}, 2000);

CSS Lazy Loading in AngularJS

_config.yml

The simple way

A classical Angular app starts with a HTML file. In the head tag you’ll put all the link tags for your templates style, and under the body tag you’ll stick the ng-app directive.

The Problem

  1. Every module should be an atomic unit: Each module (controller,template and a CSS) should function as an independent unit. We should be able to take a module out and add new one without major changes to the whole app.

In a classical angular app, with each module template taking in out out you should maintain the link tags in the Head. With many templates this is no a pretty sight…

The right way to too it would be adding the link tag inside the module template file. The problem is that link tag is not allowed inside the body tag, and although it will work, it ain’t going to be pretty.

  1. A template should be able to ask 3rd party style: In addition to it’s own style file, a template may use a 3rd party UI lib (like AngularUI, AngularStrap or a custom directive with a CSS file).

Maintaining this dependencies in the head tag by hand is not a big fun and error prone. In addition, multiple modules can use the same 3rd party lib, and we should make sure to load it’s CSS file only once because browsers are not smart to prevent it.

  1. Sometimes we have no access to the head tag: In our case, this application is wrapped with many JSP templates and pages, and it should be self contained. Touching the wrapping JSPs is not an option.

The first solution

I created an Angular service that can load a CSS file by appending a link tag to the head tag and I’ve using it in the controllers:

angular.module('cssLoadingService', []).factory("CssLoadingService", function () {
  return {
    loadCss: function (url) {
      if (document.createStyleSheet) {
        document.createStyleSheet(url); //IE
      } else {
        var link = document.createElement("link");
        link.type = "text/css";
        link.rel = "stylesheet";
        link.href = url;
        document.getElementsByTagName("head")[0].appendChild(link);
      }
    }
  };
});


function SomeController(CssLoadingService) {
  CssLoadingService.loadCss("style.css");
}

But this is not a good solution. First, it don’t prevent loading the 3rd party CSS files over multiple times. Second’ it is ugly! the CSS let the browser know how it should render the template. It’s place is in the template and not in the controller!

A better solution

I’ve created a new directive and used it in the templates. I added a peace of logic to verify we are loading each style file only once:

app.directive('lazyStyle',
  function () {
    var loadedStyles = {};
    return {
      restrict: 'E',
      link: function (scope, element, attrs) {

        var stylePath = attrs.href;

        if (stylePath in loadedStyles) {
          return;
        }

        if (document.createStyleSheet) {
          document.createStyleSheet(stylePath); //IE
        } else {
          var link = document.createElement("link");
          link.type = "text/css";
          link.rel = "stylesheet";
          link.href = stylePath;
          document.getElementsByTagName("head")[0].appendChild(link);
        }

        loadedStyles[stylePath] = true;

      }
    };
  });

// In the template:
<lazy-style href="style.css"/>

Calculated paths

In my app, we are using a server mechanism to calculate the path to all of our resources, so the href value should be calculated:

<lazy-style href="style/myModule/style.css"/>

The problem is that the directive code is executed before the basicResourcePath is being evaluated which leads to a wrong CSS path. The solution I’ve found is to use the ``$observe` on the attributes to the directive:

app.directive('lazyStyle',
  function () {
    var loadedStyles = {};
    return {
      restrict: 'E',
      link: function (scope, element, attrs) {

        attrs.$observe('href', function (value) {

          var stylePath = value;

          if (stylePath in loadedStyles) {
            return;
          }

          if (document.createStyleSheet) {
            document.createStyleSheet(stylePath); //IE
          } else {
            var link = document.createElement("link");
            link.type = "text/css";
            link.rel = "stylesheet";
            link.href = stylePath;
            document.getElementsByTagName("head")[0].appendChild(link);
          }

          loadedStyles[stylePath] = true;

        });
      }
    };
  });

link to the original post with comments