This article describes a new algorithm for the processing modeler called feature filter algorithm. If you are already familiar with ETL concepts and the graphical modeler, you can directly jump to the section the feature filter algorithm.

Building workflows for repetitive tasks

When building workflows for simple or complex geodata infrastructures, one of the most common tasks one encounters is to extract some of the features and copy them to another destination. Sometimes they need to be modified and a few attributes calculated or deleted, maybe even the geometry needs to be adjusted or in some fancy situations one even wants to generate a couple of objects from one input object. This process is often called ETL (Extract, Transform, Load) and it is something that is worth mastering as a GIS expert.
Let’s imagine a situation where we sent a field worker out to collect information about public infrastructure, equipped with a brand-new tablet and the latest and greatest version of QField. To make his task super easy, we prepare one single layer for him with an attribute type which can be set to Bus Station, Car Parking or Train Station. Now back in the office we want to integrate this back into our spatially enabled database which has been designed with 3 target tables.
 
 

Easy enough to go to QGIS and select those features by type one after the other and use a bit of copy-paste. And maybe fiddling a bit with the attributes. But hey, after all we are a bit lazy and on the one hand like to have an ice cream later on that afternoon and on the other hand like to avoid errors. Those who are lucky enough to know SQL and have full access to the database are well enough equipped to do the job.

Short introduction to the graphical modeler

For those who just want to quickly do this job visually within QGIS, there is a tool called modeler in the processing plugin. With the help of this tool it is straightforward for everyone to automate processes. To get started with the modeler, simply enable the processing plugin and click on Processing > Graphical Modeler.
Within the modeler, there are Inputs and Algorithms available. Inputs are there to define variables, algorithms on the other hand transform those variables. In its most simple form, there is one vector feature source (a layer) as input and one algorithm, for example a fixed distance buffer which in turn has one output layer with all buffered features. Such a model can be saved and reused. To run a model directly from the modeler, click the play button on top. Once saved, it appears in the processing toolbox. Every time a model is run, the input layer can be handed to the model. Or it can even run in batch mode on a list of layers or files.
With this in place, the job of doing the buffer can now be run on 200 input layers without any manual interaction. Simple as that.
Pro tip: processing models do not have to be complex. They can also be used to preconfigure single algorithms so when an algorithm is run, the parameters which you never change are predefined already. For example you can add a Simplify geometries to 1 meter algorithm which only takes a layer as parameter and has the 1 meter tolerance built-in.

The feature filter algorithm

Now back to the job of splitting the infrastructure layer into 3 different layers. To do this job visually and easily within QGIS, there is now a new algorithm available in QGIS 3.2. It is called Feature Filter and available in the processing modeler.
To make use of it, we open the processing modeler and first add a new Vector Features input and name it Infrastructure. Since we know in this project we will always deal with points, we can make already specify that in this first dialog.
 

Let’s now add a Feature Filter algorithm and use the following configuration:
The Infrastructure layer is set as input, and we define three outputs for Train Stations, Bus Stations and Car Parking. All layers will be final outputs on which no further transformations will be applied within this model and they will be directly written to a new layer.
 

Now it’s time to run our new model and check that it does what it promised.
We can also uncheck the final output checkbox and send filtered features to further processing algorithms. For example sending them through a buffer based on an attribute size (although as a QGIS professional you know you should rather be using styles than modifying the geometry in most situations in such cases).

Conclusion

With this new algorithm built directly inside the core of QGIS, the processing framework is now able to transform and refine features of a dataset with the same precision as an open heart surgery.
Of course you can get more creative in the filter criteria.
Apart from the obvious ones to do geometry modifications, there are two particularly interesting ones if you liked this one

  • The Refactor Fields algorithm allows calculating new fields or rename fields based on expressions
  • The Append plugin allows adding those features to an existing vector layer such as a database table

The data from this walkthrough is available for download as [download id=”3917″].
If you would like to test this new feature but do not yet have a concrete use-case in mind, here is a task for you: get an openstreetmap extract, import it using ogr2ogr and split the lines into different layers roads, rivers and railways, the polygons into lakes, forests and cities, the points according to your own liking.
If there is big enough interest for this, we might write another blog post on this topic.
 
We would like to thank the QGIS user group Switzerland for making this project possible through funding.
 


10 Comments

Eckart · 2018-05-28 at 13:44

Nice feature. How to do that separate with every feature in one layer full of overlapping polygons? How to build such a filter: for every id make one layer?

    Matthias Kuhn · 2018-05-28 at 14:04

    Hi Eckart,
    This new algorithm focuses on creating a predefined number of layers (one per filter) and cannot produce a variable number of outputs depending on the number of features or other layer properties.
    Depending on what problem exactly you are trying to solve, it might also be possible that what you are looking for is a custom rendering order or similar measures.

Tony Walters · 2018-06-02 at 15:30

A use case for my work would be to input an externally acquired file-based shapefile or KMZ structure, and then to perform a series of steps with output being a layer.
In my case, the steps might look like this:
Read input polygon shapefile from filesystem
Create temporary layer in a specific layer group
Apply dissolve algorithm on temp layer
Apply style parameters, CRS etc.
Save as persistent layer
Remove temporary layers
Finish
This would automate the process of external data acquisition and reduce errors.
Can it be done?
Regards,
Tony

    Matthias Kuhn · 2018-06-02 at 18:13

    This sounds like exactly the kind of work that the processing modeler can streamline.
    Make a new model with a vector layer (polygon) input, connect it to a dissolve algorithm and subsequent other ones as desired, as the very last step apply the layer styling via yet another algorithm.
    When you start this model, you will be asked for the input layer and for a final destination for the output layer.
    There will be no temporary layers in a specific layer group, the intermediate process will be completely transparent to the user.

      Tony Walters · 2018-12-25 at 19:15

      Wonderful. Have begun this and should save many hours per project. These projects include running rural broadband wireless coverage models and through a rest api import and layer coverage polygons over digital surface models (e.g. Lidar) terrain and rural housing locations. I can then count dwellings that are covered and iteratively narrow down the locations for transmission towers.

Johanness J · 2018-09-19 at 08:06

Thank you for the wonderful article, I would not have known that this feature has been incorporated otherwise. I have used the Processing Modeler in QGIS 2.18.15 in the past, and had to manually filter features using the Right Click > Filter > Expression > Save As for sub-setting different features within a Shapefile.
I can see how this new Feature Filter speeds up the previously cumbersome method I used to perform this task. My question is; can we implement this function alongside other algorithms (e.g. Clip Raster by Mask Layer)? Where it becomes more like a Batch Process of a specified algorithm using the filtered features from ‘Feature Filter’
Because, even with this new Feature Filter, I would still have to individually complete the model by linking each of the Non-final outputs from the Feature Filter as identical processes within the processing modeler. E.g. if I have 3 rasters, and 1 Shapefile, and that Shapefile had 10 polygons within it, I will be able to use the Feature Filter to filter out each of the 10 polygons, but if i want to connect those filtered outputs to other algorithms (which I want to execute across all 10 polygons), I would still need to do those individually to complete the model
Do correct me if I pose any misunderstanding, thank you! Hope to hear from you soon

    Matthias Kuhn · 2018-09-19 at 10:27

    This sounds like an interesting request, but I am not yet sure I can completely follow you.
    As an example, what you want to do is:

    • 10 filter conditions (each one returning a set of polygons)
    • clip a raster file for each of these 10 conditions
    • as a result have 10 clipped rasters

    This would be a very interesting other building block for models: the possibility to loop over a set of feature sources (algorithm ouptuts) or features (the combined polygons of all features in a source) and run another algorithm with each of them as input.

GJ · 2019-11-08 at 11:40

Thanks for the description!
Is it possible to use an user input (text or number), analogous to the Input layer in your example, as a basis for the Filter Expression?

    mkuhn · 2019-11-11 at 11:17

    Hi GJ,
    You can use pretty much everything as input. What’s the use case you are tackling?

El Moussaoui · 2022-02-14 at 10:18

Hello,

Thank you for this article !
I have a question ! How to filter my data from a selected entity?
For example, for a hand-selected school, I want my layer ‘students’ to be filtered based on those studying at the selected school.

Thanks in advance,
Youssra

Leave a Reply to EckartCancel reply