Orion: A next-generation automation testing tool

https://wesovilabs-tools.github.io/orion

Orion is born to help us to write and automate our acceptance tests. It provides a DSL inspired by Gherkin and based in HCL.

The goal of Orion, is to provide people without technical skills a tool to write acceptance tests.

In this article, we will go through some real scenarios to learn how to deal with Orion.

Setup Orion

Showcase I: math operations

Initial approach

  • 10 + 5 = 15
  • 10 -5 = 5
# feature-math-operations.hcldescription = <<EOF
This feature is used to demonstrate that both add and subs
operations work as expected.
EOF
scenario "operation add" {
given "the variables x and y" {
set x {
value = 10
}
set y{
value = 5
}
}
when "values are added" {
set result {
value = x + y
}
print {
msg = "${x} + ${y} is ${result}"
}
}
then "the result of the operation is the expected" {
assert {
assertion = result==15
}
}
}
scenario "operation substract" {
given "variables x and y" {
set x {
value = 10
}
set y{
value = 5
}
}
when "subtract y to x" {
set result {
value = x - y
}
print {
msg = "${x} - ${y} is ${result}"
}
}
then "the result of the operation is the expected" {
assert {
assertion = result==5
}
}
}

Let’s executeorion run --input feature-math-operations.hcl

Hooks

We’ll use the special hook after each to print the operation result. As a consequence, we can remove the action printin the section when from the scenarios. The hook will be executed after any scenario is completed.

# feature-math-operations.hcldescription = <<EOF
This feature is used to demonstrate that both add and subs
operations work as expected.
EOF
after each {
print {
msg = "the output of this operation is ${result}"
}
}
scenario "operation add" {
given "the variables x and y" {
set x {
value = 10
}
set y{
value = 5
}
}
when "values are added" {
set result {
value = x + y
}
}
then "the result of the operation is the expected" {
assert {
assertion = result==15
}
}
}
scenario "operation substract" {
given "variables x and y" {
set x {
value = 10
}
set y{
value = 5
}
}
when "subtract y to x" {
set result {
value = x - y
}
}
then "the result of the operation is the expected" {
assert {
assertion = result==5
}
}
}

Let’s executeorion run --input feature-math-operations.hcl

Input variables

# feature-math-operations.hcldescription = <<EOF
This feature is used to demonstrate that both add and subs
operations work as expected.
EOF
input {
arg x {
default = 10
}
arg y {
default = 5
}
arg sumResult {
default = 15
}
arg subResult {
default = 5
}
}
after each {
print {
msg = "the output of this operation is ${result}"
}
}
scenario "operation add" {
when "values are added" {
set result {
value = x + y
}
}
then "the result of the operation is the expected" {
assert {
assertion = result==sumResult
}
}
}
scenario "operation substract" {
when "subtract y to x" {
set result {
value = x - y
}
}
then "the result of the operation is the expected" {
assert {
assertion = result==subResult
}
}
}

Since we defined default values for all the args we could just run the features as usual:orion run --input feature-math-operations.hcl

On the other hand, we can set values for variables, as’s shown below

# variables-math-operations.hclx = 54
y = 46
sumResult = 100
subResult = 8

Let’s executeorion run --input feature-math-operations.hcl --vars vars-math-operations.hcl

Skip scenarios

# feature-math-operations.hcldescription = <<EOF
This feature is used to demonstrate that both add and subs
operations work as expected.
EOF
input {
arg x {
default = 10
}
arg y {
default = 5
}
arg sumResult {
default = 15
}
arg subResult {
default = 5
}
}
after each {
print {
msg = "the output of this operation is ${result}"
}
}
scenario "operation add" {
when "values are added" {
set result {
value = x + y
}
}
then "the result of the operation is the expected" {
assert {
assertion = result==sumResult
}
}
}
scenario "operation substract" {
ignore = x > 10
when "subtract y to x" {
set result {
value = x - y
}
}
then "the result of the operation is the expected" {
assert {
assertion = result==subResult
}
}
}

If we don’t pass the variables file both scenarios will be executed because the default value of x is 10. On the other hand, if we pass the variables file, the second scenario won’t be executed because x is 54.

Includes

# scenario-sum.hcl
scenario "operation add" {
when "values are added" {
set result {
value = x + y
}
}
then "the result of the operation is the expected" {
assert {
assertion = result==sumResult
}
}
}
# scenario-sub.hcl
scenario "operation substract" {
ignore = x > 10
when "subtract y to x" {
set result {
value = x - y
}
}
then "the result of the operation is the expected" {
assert {
assertion = result==subResult
}
}
}

and then, we just make use of attribute includes

# feature-math-operations.hcldescription = <<EOF
This feature is used to demonstrate that both add and subs
operations work as expected.
EOF
input {
arg x {
default = 10
}
arg y {
default = 5
}
arg sumResult {
default = 15
}
arg subResult {
default = 5
}
}
after each {
print {
msg = "the output of this operation is ${result}"
}
}
includes = [
"scenario-sum.hcl",
"scenario-sub.hcl"
]

Let’s executeorion run --input feature-math-operations.hcl

Conditional actions

# scenario-sub.hcl
scenario "operation substract" {
when "subtract y to x" {
set result {
value = x - y
when = x > y
}
set result {
value = y - x
when = x <= y
}
}
then "the result of the operation is the expected" {
assert {
assertion = result==subResult
}
}
}

if we swap the values of x and y in file vars-math-operations.hcl

# variables-math-operations.hclx = 46 
y = 54
sumResult = 100
subResult = 8

the execution of our scenarios will be successful

Let’s executeorion run --input feature-math-operations.hcl --vars vars-math-operations.hcl

Multiple cases

# scenario-sub.hcl
scenario "operation substract" {
examples = [
{ x = 20, y = 10, subResult= 10},
{ x = 10, y = 20, subResult= 10},
{ x = 5, y = 5, subResult= 0},
]
when "subtract y to x" {
set result {
value = x - y
when = x > y
}
set result {
value = y - x
when = x <= y
}
}
then "the result of the operation is the expected" {
assert {
assertion = result==subResult
}
}
}

if we execute Orion, the above scenario will be executed three times.

Let’s executeorion run --input feature-math-operations.hcl

Special action: block

# scenario-mult.hcl
scenario "operation multiplication" {
examples = [
{ x = 20, y = 10, multResult= 10},
{ x = -1, y = -2, multResult= 2},
{ x = 5, y = 5, multResult= 25},
{ x = 5, y = 0, multResult= 0},
]
when "multiply y by x" {
set result {
value = x * y
}
}
then "the result of the operation is the expected" {
assert {
assertion = result==multResult
}
}
}

but I want to show you how the special action block (Documentation here) works. This action is used to group a set of actions. And additionally, we will make use of an attribute count that permits iterate from 0 to count-1.

# scenario-mult.hcl
scenario "operation multiplication" {
examples = [
{ x = 20, y = 10, multResult= 200},
{ x = -1, y = -2, multResult= 2},
{ x = 5, y = 5, multResult= 25},
{ x = 5, y = 0, multResult= 0},
]
given "initialie result" {
set result {
value = 0
}
}
when "multiply y by x" {
block {
set result {
value = result + x
}
print {
msg = "${x} * ${_.index+1} is ${result}"
}
count = y
when = x>0 && y>0
}
set result {
value = x * y
when = x<0 || y<0
}
}
then "the result of the operation is the expected" {
assert {
assertion = result==multResult
}
}
}

And of course, we need to add this file in the include block.

# feature-math-operations.hcldescription = <<EOF
This feature is used to demonstrate that both add and subs
operations work as expected.
EOF
input {
arg x {
default = 10
}
arg y {
default = 5
}
arg sumResult {
default = 15
}
arg subResult {
default = 5
}
}
after each {
print {
msg = "the output of this operation is ${result}"
}
}
includes = [
"scenario-sum.hcl",
"scenario-sub.hcl",
"scenario-mult.hcl"
]

Let’s executeorion run --input feature-math-operations.hcl

Beside of when and count there are other special attributes. See the full list here.

Orion is still in beta, hence your suggestions and feedback will be very appreciated!

Links of interests