Unresolved reference while trying to import col from pyspark.sql.functions in python 3.5

Try installing 'pyspark-stubs', I had the same problem in PyCharm and by doing it I resolved it.

It turns out to be IntelliJ IDEA's problem. Even though it shows unresolved reference, my program still runs without any problem in the command line.

The function like col is not an explicit function defined in python code, but rather dynamically generated.

It will also report an error by static analysis tool like pylint

So the easiest way to use it shall be something like this

from pyspark.sql import functions as F

F.col("colname")

The following code in python/pyspark/sql/functions.py

_functions = {
    'lit': _lit_doc,
    'col': 'Returns a :class:`Column` based on the given column name.',
    'column': 'Returns a :class:`Column` based on the given column name.',
    'asc': 'Returns a sort expression based on the ascending order of the given column name.',
    'desc': 'Returns a sort expression based on the descending order of the given column name.',

    'upper': 'Converts a string expression to upper case.',
    'lower': 'Converts a string expression to upper case.',
    'sqrt': 'Computes the square root of the specified float value.',
    'abs': 'Computes the absolute value.',

    'max': 'Aggregate function: returns the maximum value of the expression in a group.',
    'min': 'Aggregate function: returns the minimum value of the expression in a group.',
    'count': 'Aggregate function: returns the number of items in a group.',
    'sum': 'Aggregate function: returns the sum of all values in the expression.',
    'avg': 'Aggregate function: returns the average of the values in a group.',
    'mean': 'Aggregate function: returns the average of the values in a group.',
    'sumDistinct': 'Aggregate function: returns the sum of distinct values in the expression.',
}

def _create_function(name, doc=""):
    """ Create a function for aggregator by name"""
    def _(col):
        sc = SparkContext._active_spark_context
        jc = getattr(sc._jvm.functions, name)(col._jc if isinstance(col, Column) else col)
        return Column(jc)
    _.__name__ = name
    _.__doc__ = doc
    return _

for _name, _doc in _functions.items():
    globals()[_name] = since(1.3)(_create_function(_name, _doc))