build & release pipeline

Test Automation Day 2019

Two weeks ago I went to the Test Automation Day 2019 conference in Utrecht, together with my colleague Diego Fung-Loy (Senior Test & Requirements Analyst). Here are some of the take-aways from this inspiring conference.

(I love the number on my badge: 242)

09.30 - Keynote: Balancing your test automation with exploratory testing

The opening keynote by Mirjana Kolarov (from Novi Sad, Republic of Serbia) had “Learn & Explore” as the motto.

  • Beware of inattentional blindness (the gorilla can be a huge bug).
  • Exploratory testing is like a question game (“is it alive? is it …”) so the first question often determines how quick you get to the solution.
  • For E2E (end-to-end) testing you shouldn’t test everything, just the the flows that are important to your business.
  • The classical test automation pyramid with a wide Unit Test base, followed by a narrower Integration Tests layer and a small E2E top might look different for your business: like a diamond or even an inverted pyramid with lots of E2E testing and few unit tests.
  • Controversial statement: testers need to be able to read code to see what needs to be tested.
  • Before automating tests first do lots of manual testing to learn about the new feature.

When to use Exploratory Testing

  • Functional testing
  • Performance testing
  • Load testing
  • Stress testing
  • Security testing
  • Usability testing
  • Preparation for automated tests

When to use Automation In Testing (= better term than “Automated Testing”)

  • Regression testing
  • Smoke testing
  • Data driven testing
  • Load testing
  • Security testing
  • CI and CD

She ended here presentation with a GIF showing the experience of tester vs client, which inspired me to look for one myself:

Which may be funny but is of course not a good representation of the average Nextens customer, who are more of the athletic type as demonstrated by this recent marketing video:

10.10 - Keynote: Continuously Shipping Web Apps with Confidence

Amir Rustamzadeh - senior engineer for JavaScript End to End Testing Framework Cypress - showed me that everything that we struggle with in Selenium might be solved by Cypress!

The web went from document viewer to application platform, but the test tools did not adept to a web that went from stateless to stateful.

Because testing is hard it’s important to remove friction.

Some features of Cypress:

  • Reliable in-browser testing
  • Free, open source
  • “Makes testing an enjoyable experience, and generated developer happiness”
  • > npm install -D cypress
  • API is English-like, chainable and easy to read
  • Offers reusable custom commands
  • Where Selenium queries the browser Cypress lives inside the browser, with a Node.js backend
  • Faster tests
  • Direct native access to the DOM and your application enables new types of testing
  • You can login to your app without using the form - use a login function instead
  • Offers automatic waiting
  • Promises flake free testing (no false negatives)
  • Via the backend you can execute Javascript on the system to clear a database, create users or seed a db
  • You can run Cypress headless as part of your CI with video recording out-of-the box!
  • Record results to a dashboard website
  • Supports parallelization (to optimize CI usage)

If you make testing simple enough people will start writing tests!

11.20 - Breakout Session: The Story of the Little Jenkinsfile that Could

Shelley Lambert is the Test Lead for IBM Runtime Technologies team and told about her journey from inheriting legacy test code to running 87 million tests every night.

Some principles:

  • reduce friction: fewer dependencies, files and locations requiring change
  • restrict new test features to only those that drive next actions

Human Challenges:

  • Feature creep
  • Ownership & access
  • Technical ego
  • Naysayers

Technical Challenges:

  • Open tools churn (updates, defects, skew)
  • Unstable machine layer

These sessions did not generate any notes worth reproducing here

  • 12.10 - Verifying micro services in chains without chain testing?
  • 14.00 - Keynote: Will robots replace testers?
  • 14.50 - Installing a test automation infrastructure at a Swiss bank
  • 16.00 - Why Automated Performance Testing is a must (I did not write down this consultant’s advice to hire a professional consultant when implementing automated performance testing)

16.50 - Keynote: A Continuous Life in Test Automation

Dorothy Graham is amazing - she has been at the forefront of many discoveries in software testing (and might not have always gotten credit for it because of her gender):

Dorothy has worked in software testing for over 40 years. Originally from the US, her first job was developing testing tools for Bell Labs in New Jersey. She then worked as a developer for Ferranti Computer Systems in Manchester before moving to the National Computing Centre where she developed training courses in software engineering.

She later became an independent consultant specialising in software testing and founded Grove Consultants in 1988. Over the next 20 years, Grove gained a reputation for the high quality of their training courses, materials and presentations establishing a world-wide client base. In 2008 she left Grove to work again as an independent consultant. Dorothy now uses her experience to help educate and motivate others through her popular and entertaining conference presentations.

She is co-author of five books: an introductory book for software testing together with more in depth books on Software Inspection and Test Automation.Recently she has worked with Seretta Gamba developing a Wiki of software testing patterns. Their latest book has now been published showing how patterns can be used to help solve testing problems.

Source: http://www.dorothygraham.co.uk/aboutDG/index.html

This was (probably) her final keynote on the continent - some quotes:

  • Be open to the unexpected
  • Writing things that are shared around is great
  • Asking questions is very powerful
  • Forgiveness is better than permission - if you have a good idea just do it!
  • Automating chaos is just faster chaos
  • Managers don’t understand testing
  • Continuous everything
  • Danger: pressure for testers to learn coding

The essence of testing:

  • What could go wrong?
  • What if it isn’t?
  • Have you thought of…
  • Asking “stupid” questions