No output in PyQGIS standalone script running Distance Matrix?

I have found and fixed the final errors. I tried changing how each algorthim parameter got its value.

By changing the code below:


inputfield = QgsProperty.fromField("InputPointID")
targetfield = QgsProperty.fromField("TargetPointID")

To this:


inputfield = QgsProperty.fromValue("InputPointID")
targetfield = QgsProperty.fromValue("TargetPointID")

It has fixed the final problems and the script runs well.

My final working example of the PYQGIS 3 standalone code (Including all updates described above) to run the QGIS 3 Distance Matrix in the OS shell is as below. For info/interest, I will be tweaking it next to run in GNU Parallel; I can do that bit so I won't take up any more time on this question. Thanks to everyone. Question closed.



######### INITIALISE QGIS STANDALONE ################

import sys
from qgis.core import (
     QgsApplication, 
     QgsProcessingFeedback, 
     QgsVectorLayer,
     QgsField,
     QgsFields,
     QgsProperty,
     QgsProcessingFeatureSourceDefinition,
     QgsProcessingOutputLayerDefinition
)

#start QGIS instance without GUI
#Make sure the prefix is correct. Even though QGIS is in '/usr/share/qgis',
#the prefix needs to be '/usr' (assumes Debian OS)

QgsApplication.setPrefixPath('/usr', True)
myqgs = QgsApplication([], False)
myqgs.initQgis()

######### INITIALISE THE PROCESSING FRAMEWORK ################

# Append the path where processing plugin can be found (assumes Debian OS)
sys.path.append('/usr/share/qgis/python/plugins')

#import modules needed
import processing
from processing.core.Processing import Processing

#start the processing module
processing.core.Processing.Processing.initialize()

######### Set Distance Matrix plugin parameters ############################

# I used this command in the QGIS3 python console
# >>> processing.algorithmHelp("qgis:distancematrix") 
# to get the object types it accepts for each parameter

inputlayer = QgsVectorLayer('/path/to/myinputfile.shp', 'layer 1', 'ogr')
targetlayer = QgsVectorLayer('/path/to/mytargetfile.shp', 'layer 2', 'ogr')

#do an Input Layers Validity Check
print(inputlayer.isvalid())
print(targetlayer.isvalid())

inputfield = QgsProperty.fromValue("InputPointID")
targetfield = QgsProperty.fromValue("TargetPointID")

matrixtype = QgsProperty.fromValue(0)
nearestpoints = QgsProperty.fromValue(0)

outputlayer = QgsProcessingOutputLayerDefinition('/path/to/myoutputfile.csv')

params = { 
    'INPUT' : inputlayer,
    'INPUT_FIELD' : inputfield,
    'TARGET' : targetlayer,
    'TARGET_FIELD' : targetfield,
    'MATRIX_TYPE' : matrixtype,
    'NEAREST_POINTS' : nearestpoints,
    'OUTPUT' : outputlayer
}

feedback = QgsProcessingFeedback()

res = processing.run('qgis:distancematrix', params, feedback=feedback)
res['OUTPUT'] # Access your output layer

As per Joseph's request please find attached alternative version to allow the script to run in GNU Parallel. It assumes QGIS3 and Debian.

The method I have found best is to design it as follows:

  1. convert the internal file calls in the python code to arguments

  2. create an OS script wrapper (e.g. bash) to run the python code (to make sure it can be run without problems via the shell before moving to GNU Parallel)

  3. create a GNU Parallel wrapper which runs the OS shell wrapper

I have the Distance Matrix working in parallel now. So following the above method, I adjusted the python code to the below:


######### INITIALISE QGIS STANDALONE ################

import sys
from qgis.core import (
     QgsApplication, 
     QgsProcessingFeedback, 
     QgsVectorLayer,
     QgsField,
     QgsFields,
     QgsProperty,
     QgsProcessingFeatureSourceDefinition,
     QgsProcessingOutputLayerDefinition
)

#start QGIS instance without GUI
#Make sure the prefix is correct. Even though QGIS is in '/usr/share/qgis',
#the prefix needs to be '/usr' (assumes Debian OS)

QgsApplication.setPrefixPath('/usr', True)
myqgs = QgsApplication([], False)
myqgs.initQgis()

######### INITIALISE THE PROCESSING FRAMEWORK ################

# Append the path where processing plugin can be found (assumes Debian OS)
sys.path.append('/usr/share/qgis/python/plugins')

#import modules needed
import processing
from processing.core.Processing import Processing

#start the processing module
processing.core.Processing.Processing.initialize()

######### Set Distance Matrix plugin parameters ############################

# I used this command in the QGIS3 python console
# >>> processing.algorithmHelp("qgis:distancematrix") 
# to get the object types it accepts for each parameter

inputlayer = qgis.core.QgsProperty.fromValue(sys.argv[1])
targetlayer = qgis.core.QgsProperty.fromValue(sys.argv[2])

inputfield = QgsProperty.fromValue(sys.argv[3])
targetfield = QgsProperty.fromValue(sys.argv[4])

matrixtype = QgsProperty.fromValue(0)
nearestpoints = QgsProperty.fromValue(0)

outputlayer = QgsProcessingOutputLayerDefinition(sys.argv[5])

params = { 
    'INPUT' : inputlayer,
    'INPUT_FIELD' : inputfield,
    'TARGET' : targetlayer,
    'TARGET_FIELD' : targetfield,
    'MATRIX_TYPE' : matrixtype,
    'NEAREST_POINTS' : nearestpoints,
    'OUTPUT' : outputlayer
}

feedback = QgsProcessingFeedback()

res = processing.run('qgis:distancematrix', params, feedback=feedback)
res['OUTPUT'] # Access your output layer

Then I created an OS wrapper for the python script. An example bash script is below:

#!/bin/bash

########### set the arguments for the Distance Matrix script (Input File, Target File, Input File ID Field, Target File ID Field) ################################
########### this script assumes there is 1 or more input files and the same target file  for all input file(s) #########################################

firstarg=$1 
firstargfileformat=".shp"

secondarg="path/to/targetfile"
secondargfileformat=".shp"

thirdarg="InputPointID"
fourtharg="TargetPointID"

fifthargfileprefix="YourChosenPrefix"
fifthargfileformat=".csv"

######## make a copy of the target file to use for processing which matches the name of the input file, for when it runs in parallel mode, so multiple parallel threads are not fighting for access to the same file ###############

#first get the name of the input file (without its path and without its extension)
inputfilenamewithoutfilepath=$(basename -- "$firstarg")
inputfilenamewithoutextension=$(basename --suffix=$firstargfileformat "$inputfilenamewithoutfilepath")

#test it has got the file name correctly
echo $inputfilenamewithoutextension

#then get the file extension of the target file
targetfilenamewithoutfilepath=$(basename -- "$secondarg")
targetfilenamewithoutextension=$(basename --suffix=$secondargfileformat "$targetfilenamewithoutfilepath")

#create the name of the temporary target file from the name of the input file (without its extension)
combinedtargetfilename=${inputfilenamewithoutextension}$secondargfileformat
combinedtargetfilenamewithpath=TempTargets/$combinedtargetfilename

#check the target file name (with and without path) has been derived correctly
echo "Combined Target File Name: $combinedtargetfilename Combined Target File Name (with path): $combinedtargetfilenamewithpath"

#copy the file
cp $secondarg $combinedtargetfilenamewithpath

#derive the outputfilename
combinedoutputfilenamepart1=${inputfilenamewithoutextension}$fifthargfileformat
combinedoutputfilename=${fifthargfileprefix}$combinedoutputfilenamepart1
combinedoutputfilenamewithpath=path/to/$combinedoutputfilename

#check the output file name (with and without path) has been derived correctly
echo "Combined Output File Name $combinedoutputfilename Combined Output File Name with path $combinedoutputfilenamewithpath"

#########run the standalone QGIS Distance Matrix ###############

python3 pyqgisstandalonedistancematrix.py $firstarg $combinedtargetfilenamewithpath $thirdarg $fourtharg $combinedoutputfilenamewithpath

######## delete temporary files ###############

rm $combinedtargetfilenamewithpath

####### exit the script############

exit 0

Then created a GNU Parallel wrapper. Example in bash script again.


#!/bin/bash

#reset the shell SECONDS timer
SECONDS=0

#log the start datetime
startdatetime=$(date)
echo "$startdatetime"

#run the parallel script
parallel ./OS_Wrapper_for_Distance_Matrix.sh ::: Inputs/*.shp

#get the number of seconds which passed since the script started
runduration=$SECONDS

#log the finish datetime
enddatetime=$(date)
echo "$enddatetime"

#elapsedsecs=$(( $(date -d "$enddatetime" "+%s")-$(date -d "$startdatetime" "+%s") ))

printf 'This run took %dd:%dh:%dm:%ds\n' $(($SECONDS/86400)) $(($SECONDS%86400/3600)) $(($SECONDS%3600/60)) \ $(($SECONDS%60))
echo "Run started at $startdatetime"
echo "Run finished at $enddatetime"

exit 0

That's it. you can add more arguments to the python and bash scripts if you need to for your particular needs. For the job I used it for over the last couple of days it went from taking just over 4 days using pyqgis via the GUI, to 5 hours using the PYQGIS standalone script with GNU Parallel on my 8-core machine.

Tags:

Distance Matrix

Pyqgis

Qgis Processing

Qgis 3

Recent Posts