{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "e06a27bf",
   "metadata": {},
   "source": [
    "# CQL2 Filtering\n",
    "\n",
    "This notebook demonstrates using pystac-client to filter STAC items with [CQL2](https://docs.ogc.org/is/21-065r2/21-065r2.html) as described in the [STAC API Filter Extension](https://github.com/stac-api-extensions/filter). \n",
    "\n",
    "Note: Not all STAC APIs support the Filter Extension. APIs advertise conformance by including `https://api.stacspec.org/v1.0.0/item-search#filter` in the `conformsTo` attribute of the root API."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b65de617",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "import geopandas as gpd\n",
    "import pandas as pd\n",
    "\n",
    "from pystac_client import Client"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c8ac88bb",
   "metadata": {},
   "source": [
    "The first step as always with pystac-client is opening the catalog:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "98942e75",
   "metadata": {},
   "outputs": [],
   "source": [
    "# STAC API root URL\n",
    "URL = \"https://planetarycomputer.microsoft.com/api/stac/v1\"\n",
    "\n",
    "catalog = Client.open(URL)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e16077c",
   "metadata": {},
   "source": [
    "## Initial Search Parameters\n",
    "\n",
    "Here we set up some initial search parameters to use with the `Client.search` function. We are providing a maximum number of items to return (`max_items`), a collection to look within (`collections`), a geometry (`intersects`), and a datetime range (`datetime`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5e961981",
   "metadata": {},
   "outputs": [],
   "source": [
    "# AOI around Delfzijl, in the north of The Netherlands\n",
    "geom = {\n",
    "    \"type\": \"Polygon\",\n",
    "    \"coordinates\": [\n",
    "        [\n",
    "            [6.42425537109375, 53.174765470134616],\n",
    "            [7.344360351562499, 53.174765470134616],\n",
    "            [7.344360351562499, 53.67393435835391],\n",
    "            [6.42425537109375, 53.67393435835391],\n",
    "            [6.42425537109375, 53.174765470134616],\n",
    "        ]\n",
    "    ],\n",
    "}\n",
    "\n",
    "params = {\n",
    "    \"max_items\": 100,\n",
    "    \"collections\": \"landsat-8-c2-l2\",\n",
    "    \"intersects\": geom,\n",
    "    \"datetime\": \"2018-01-01/2020-12-31\",\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d6f1dd5f",
   "metadata": {},
   "source": [
    "## Using Filters\n",
    "\n",
    "In addition to the parameters described above in the following examples we will filter by Item properties (`filter`) using CQL2-JSON. Here is a little function that does the search constructs a `GeoDataFrame` of the results and then plots `datetime` vs `eo:cloud_cover`.\n",
    "\n",
    "Remember that in this whole notebook we are only looking at STAC metadata, there is no part where we are reading the data itself."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8b26e89b",
   "metadata": {},
   "outputs": [],
   "source": [
    "def search_and_plot(filter):\n",
    "    search = catalog.search(**params, filter=filter)\n",
    "\n",
    "    gdf = gpd.GeoDataFrame.from_features(search.item_collection_as_dict())\n",
    "    gdf[\"datetime\"] = pd.to_datetime(gdf[\"datetime\"])\n",
    "    print(f\"Found {len(gdf)} items\")\n",
    "\n",
    "    gdf.plot.line(x=\"datetime\", y=\"eo:cloud_cover\", title=json.dumps(filter))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11afcc19",
   "metadata": {},
   "source": [
    "We can test out the function by passing an empty dict to do no filtering at all."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b6293c11",
   "metadata": {},
   "outputs": [],
   "source": [
    "search_and_plot({})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44d3bc04",
   "metadata": {},
   "source": [
    "## CQL2 Filters\n",
    "\n",
    "We will use `eo:cloud_cover` as an example and filter for all the STAC Items where `eo:cloud_cover <= 10%`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dfc0e759",
   "metadata": {},
   "outputs": [],
   "source": [
    "filter = {\"op\": \"<=\", \"args\": [{\"property\": \"eo:cloud_cover\"}, 10]}\n",
    "\n",
    "search_and_plot(filter)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75e835f1",
   "metadata": {},
   "source": [
    "Next let's look for all the STAC Items where `eo:cloud_cover >= 80%`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9c2f9ca1",
   "metadata": {},
   "outputs": [],
   "source": [
    "filter = {\"op\": \">=\", \"args\": [{\"property\": \"eo:cloud_cover\"}, 80]}\n",
    "\n",
    "search_and_plot(filter)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ad984bf",
   "metadata": {},
   "source": [
    "We can combine multiple CQL2 statements to express more complicated logic:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "109f673c",
   "metadata": {},
   "outputs": [],
   "source": [
    "filter = {\n",
    "    \"op\": \"and\",\n",
    "    \"args\": [\n",
    "        {\"op\": \"<=\", \"args\": [{\"property\": \"eo:cloud_cover\"}, 60]},\n",
    "        {\"op\": \">=\", \"args\": [{\"property\": \"eo:cloud_cover\"}, 40]},\n",
    "    ],\n",
    "}\n",
    "\n",
    "search_and_plot(filter)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "617c7416",
   "metadata": {},
   "source": [
    "You can see the power of this syntax. Indeed we can replace `datetime` and `intersects` from our original search parameters with a more complex CQL2 statement."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7b0dc965",
   "metadata": {},
   "outputs": [],
   "source": [
    "filter = {\n",
    "    \"op\": \"and\",\n",
    "    \"args\": [\n",
    "        {\"op\": \"s_intersects\", \"args\": [{\"property\": \"geometry\"}, geom]},\n",
    "        {\"op\": \">=\", \"args\": [{\"property\": \"datetime\"}, \"2018-01-01\"]},\n",
    "        {\"op\": \"<=\", \"args\": [{\"property\": \"datetime\"}, \"2020-12-31\"]},\n",
    "        {\"op\": \"<=\", \"args\": [{\"property\": \"eo:cloud_cover\"}, 60]},\n",
    "        {\"op\": \">=\", \"args\": [{\"property\": \"eo:cloud_cover\"}, 40]},\n",
    "    ],\n",
    "}\n",
    "search = catalog.search(max_items=100, collections=\"landsat-8-c2-l2\", filter=filter)\n",
    "\n",
    "print(f\"Found {len(search.item_collection())} items\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56503c7b",
   "metadata": {},
   "source": [
    "### CQL2 Text\n",
    "\n",
    "The examples above all use CQL2-json but pystac-client also supports passing `filter` as CQL2 text.\n",
    "\n",
    "NOTE: As of right now in pystac-client if you use CQL2 text you need to change the search HTTP method to GET."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5e8f62f5",
   "metadata": {},
   "outputs": [],
   "source": [
    "search = catalog.search(**params, method=\"GET\", filter=\"eo:cloud_cover<=10\")\n",
    "\n",
    "print(f\"Found {len(search.item_collection())} items\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b865c1f",
   "metadata": {},
   "source": [
    "Just like CQL2 json, CQL2 text statements can be combined to express more complex logic:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c06f40cf",
   "metadata": {},
   "outputs": [],
   "source": [
    "search = catalog.search(\n",
    "    **params, method=\"GET\", filter=\"eo:cloud_cover<=60 and eo:cloud_cover>=40\"\n",
    ")\n",
    "\n",
    "print(f\"Found {len(search.item_collection())} items\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "35cbf612",
   "metadata": {},
   "source": [
    "## Queryables\n",
    "\n",
    "pystac-client provides a method for accessing all the arguments that can be used within CQL2 filters for a particular collection. These are provided as a json schema document, but for readability we are mostly interested in the names of the fields within `properties`.\n",
    "\n",
    "NOTE: When getting the collection, you might notice that we use \"landsat-c2-l2\" as the collection id rather than \"landsat-8-c2-l2\". This is because \"landsat-8-c2-l2\" doesn't actually exist as a collection. It is just used in some places as a collection id on items. This is likely a remnant of some former setup in the Planetary Computer STAC."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "90f1cc6d",
   "metadata": {},
   "outputs": [],
   "source": [
    "collection = catalog.get_collection(\"landsat-c2-l2\")\n",
    "queryables = collection.get_queryables()\n",
    "\n",
    "list(queryables[\"properties\"].keys())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c407ffec",
   "metadata": {},
   "source": [
    "## Read More\n",
    "\n",
    "- For more involved CQL2 examples in a STAC context read the [STAC API Filter Extension Examples](https://github.com/stac-api-extensions/filter?tab=readme-ov-file#examples)\n",
    "\n",
    "- For examples of all the different CQL2 operations take a look at the [playground on the CQL2-rs docs](https://developmentseed.org/cql2-rs/latest/playground/)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
