close

How to run bash script file in Airflow

Hello Guys, How are you all? Hope You all Are Fine. Today We Are Going To learn about How to run bash script file in Airflow in Python. So Here I am Explain to you all the possible Methods here.

Without wasting your time, Let’s start This Article.

Table of Contents

How to run bash script file in Airflow?

  1. How to run bash script file in Airflow?

    From the tutorial this is OK:
    t2 = BashOperator( task_id='sleep', bash_command='sleep 5', retries=3, dag=dag)

  2. run bash script file in Airflow

    From the tutorial this is OK:
    t2 = BashOperator( task_id='sleep', bash_command='sleep 5', retries=3, dag=dag)

Method 1

From the tutorial this is OK:

t2 = BashOperator(
    task_id='sleep',
    bash_command='sleep 5',
    retries=3,
    dag=dag)

But you’re passing a multi-line command to it

create_command = """
 ./scripts/create_file.sh
"""

should be

create_command = "./scripts/create_file.sh "

Moreover, you also have to make sure that you are in the correct directory to avoid cryptic errors. Do it like this for example:

create_command = "./scripts/create_file.sh "
if os.path.exists(create_command):
   t1 = BashOperator(
        task_id= 'create_file',
        bash_command=create_command,
        dag=dag
   )
else:
    raise Exception("Cannot locate {}".format(create_command))

Method 2

From the documentation:

t2 = BashOperator(
    task_id='bash_example',
    # "scripts" folder is under "/usr/local/airflow/dags"
    bash_command="scripts/test.sh",
    dag=dag)

Conclusion

It’s all About this issue. Hope all Methods helped you a lot. Comment below Your thoughts and your queries. Also, Comment below which Method worked for you? Thank You.

Also, Read