close

[Solved] Cannot find col function in pyspark

Hello Guys, How are you all? Hope You all Are Fine. Today I get the following error Cannot find col function in pyspark in python. So Here I am Explain to you all the possible solutions here.

Without wasting your time, Let’s start This Article to Solve This Error.

How Cannot find col function in pyspark Error Occurs?

Today I get the following error Cannot find col function in pyspark in python.

How To Solve Cannot find col function in pyspark Error ?

  1. How To Solve Cannot find col function in pyspark Error ?

    To Solve Cannot find col function in pyspark Error As explained above, pyspark generates some of its functions on the fly, which makes that most IDEs cannot detect them properly.

  2. Cannot find col function in pyspark

    To Solve Cannot find col function in pyspark Error As explained above, pyspark generates some of its functions on the fly, which makes that most IDEs cannot detect them properly.

Solution 1

As explained above, pyspark generates some of its functions on the fly, which makes that most IDEs cannot detect them properly. However, there is a python package pyspark-stubs that includes a collection of stub files such that type hints are improved, static error detection, code completion, … By just installing with

pip install pyspark-stubs==x.x.x

(where x.x.x has to be replaced with your pyspark version (2.3.0. in my case for instance)), col and other functions will be detected, without changing anything at your code for most IDEs (Pycharm, Visual Studio Code, Atom, Jupyter Notebook, …)

Solution 2

In Pycharm the col function and others are flagged as “not found”

enter image description here

a workaround is to import functions and call the col function from there.

for example:

from pyspark.sql import functions as F
df.select(F.col("my_column"))

Summery

It’s all About this issue. Hope all solution helped you a lot. Comment below Your thoughts and your queries. Also, Comment below which solution worked for you? Thank You.

Also, Read