DJ Adams

Local-first dev with CAP Node.js - mocking remote services

2026-05-13T00:00:00Z

This post is one of a series on local-first development with CAP Node.js.

Calesi and mocking remote services

In the context of CAP-Level Service Integration (aka "Calesi") we can mash up remote and local services, and in the full spirit of CAP generally, run everything in airplane mode, i.e. fully locally.

Working through an example

In this post, we'll work through an example of mocking a remote service, based on content in the remoteservice/ directory of the repo set up for the related talk.

We start with an almost empty project directory, save for a basic package.json file which we only really have at this point so we can check the changes in it that are introduced when we import a remote service API definition. All operations are done in the context of this project directory.

The other files in this directory in the repo are related to performing and resetting the demo during the talk version of this post.

Examining and importing the candidate remote service

At https://odd.cfapps.eu10.hana.ondemand.com/ there's the northbreeze OData V4 service with Products, Categories and Suppliers¹. The service's API is available as the metadata document so let's grab that:

curl \
  --url 'https://odd.cfapps.eu10.hana.ondemand.com/northbreeze/$metadata' \
> northbreeze.edmx

and use cds import to import it to our project:

cds import northbreeze.edmx

This emits:

[cds] - updated ./package.json

[cds] - imported API to srv/external/northbreeze
> use it in your CDS models through the likes of:

using { northbreeze as external } from './external/northbreeze';

and the EDMX API definition, along with the CAP-focused CSN equivalent that was created at import, are moved into an external/ directory within a standard srv/ directory (which itself is autovivified at this point):

./
├── package.json
└── srv/
    └── external/
        ├── northbreeze.csn
        └── northbreeze.edmx

Mocking is automatically and immediately initiated

At this point too, the CAP server restarts and shows:

[cds] - loaded model from 1 file(s):

  srv/external/northbreeze.csn

[cds] - using bindings from: { registry: '~/.cds-services.json' }
[cds] - connect to db > sqlite { url: ':memory:' }
/> successfully deployed to in-memory database.

[cds] - mocking northbreeze {
  at: [ '/odata/v4/northbreeze' ],
  decl: 'srv/external/northbreeze.csn:170'
}
[cds] - server listening on { url: 'http://localhost:4004' }

This is because cds watch is actually shorthand for:

cds serve all --with-mocks --in-memory?

and if we read the help for --with-mocks we see this:

Use this in combination with the variants serving multiple services.

It starts in-process mock services for all required services configured in package.json#cds.requires, which don't have external bindings in the current process environment.

Note that by default, this feature is disabled in production and must be enabled with configuration 'features.mocked_bindings=true'.

That's right - mocking is already being done for our imported remote service! It's in-process, i.e. within the same CAP server process that we started with cds watch.

The package.json file is extended

As part of the import process, package.json was modified in two key areas:

--- a/remoteservice/package.json
+++ b/remoteservice/package.json
@@ -4,7 +4,10 @@
   "description": "Demonstrating local-first support for remote services",
   "dependencies": {
     "@sap/cds": "^9",
-    "express": "^4"
+    "express": "^4",
+    "@sap-cloud-sdk/connectivity": "^4",
+    "@sap-cloud-sdk/http-client": "^4",
+    "@sap-cloud-sdk/resilience": "^4"
   },
   "devDependencies": {
     "@cap-js/sqlite": "^2"
@@ -12,5 +15,13 @@
   "scripts": {
     "start": "cds-serve"
   },
-  "private": true
-}
+  "private": true,
+  "cds": {
+    "requires": {
+      "northbreeze": {
+        "kind": "odata",
+        "model": "srv/external/northbreeze"
+      }
+    }
+  }
+}

the SAP Cloud SDK libraries were added - for marshalling of, connection to and communication with remote destinations²
the remote service is added as "required", with the name northbreeze

Reviewing the situation

This in-process mocking of the required remote service "northbreeze" means that in the CAP server context that exists for our project, we have that remote service available to us:

But while the mocked remote service is already fully formed, even in this "in-process" mode, there's no data. Let's add some so we can better explore the service.

Add data for the mocked remote service

Using CAP's mock data facilities, we can easily come up with some mock data. Because of CAP's convention over configuration axiom, this works even for mocked remote services. As an erstwhile Perl programmer, I appreciate this DWIM-style approach.

Have some data generated

Let's first have CAP generate some data for us:

cds \
  add data \
  --filter Categories \
  --records 10

This creates a CSV file with an appropriate name and in the expected place for initial data:

./
├── db/
│   └── data/
│       └── northbreeze.Categories.csv
├── package.json
└── srv/
    └── external/
        ├── northbreeze.csn
        └── northbreeze.edmx

Retrieve data and use that too

Given this is about gathering some data to exercise the mocked remote service, it's likely that the actual remote service has data that we can perhaps use too.

Being an OData V4 service, the data available, in the form of, say, an entityset, is going to be in JSON format by default. But that's fine, the CAP server's data mechanism can deal with this too. So let's grab the Products data from the actual remote service and place it alongside the Categories data we have:

curl \
  --silent \
  --url 'https://odd.cfapps.eu10.hana.ondemand.com/northbreeze/Products' \
| jq .value \
> db/data/northbreeze.Products.json

We can mix CSV and JSON data with ease:

./
├── db/
│   └── data/
│       └── northbreeze.Categories.csv
:       └── northbreeze.Products.json

and it's picked up as we would hope:

[cds] - connect to db > sqlite { url: ':memory:' }
  > init from db/data/northbreeze.Products.json
  > init from db/data/northbreeze.Categories.csv
/> successfully deployed to in-memory database.

Switching to a separate mocking process

So far the required remote service has been mocked in-process.

But for local development with a scenario that is closer to the eventual production scenario we can also have that service mocked in a separate process. One effect of this is that real wire API calls are made between your local service and the separately mocked (but still locally running) remote service.

Before continuing, let's stop the current CAP server.

Now let's revisit the in-process mocking, but in the context of a local to remote proxy definition for an entity. Following that, we'll then switch to separate process based mocking.

Set up a local to remote proxy definition

One of the simplest forms of service mashup is surfacing a remote entity as a local one. This may not be entirely useful, but it demonstrates the atomic structure of more involved scenarios, and is nice and simple so as not to get in the way of understanding here.

First, add the following service definition in srv/main.cds, remembering that the using directive here brings in the northbreeze scope from the imported remote service definition that was created with the cds import earlier:

using {northbreeze} from './external/northbreeze';

service Main {
  entity Products as projection on northbreeze.Products;
}

In the following sections, we'll see the difference between in-process and external process mocking.

Restart the single CAP server process

Restart the CAP server with cds watch, whereupon we will see:

[cds] - loaded model from 2 file(s):

  srv/main.cds
  srv/external/northbreeze.csn

[cds] - using bindings from: { registry: '~/.cds-services.json' }
[cds] - connect to db > sqlite { url: ':memory:' }
  > init from db/data/northbreeze.Products.json
  > init from db/data/northbreeze.Categories.csv
/> successfully deployed to in-memory database.

[cds] - serving Main {
  at: [ '/odata/v4/main' ],
  decl: 'srv/main.cds:3'
}
[cds] - mocking northbreeze {
  at: [ '/odata/v4/northbreeze' ],
  decl: 'srv/external/northbreeze.csn:170'
}

In other words:

the overall CDS model is built from the local definition we've just created, plus the definitions from the remote service
initial data is loaded from the CSV and JSON files in db/data/
the local provided service Main is served
the remote required service northbreeze is also served, mocked in-process

Moreover, when we request a Products resource from the local Main service at /odata/v4/main, which as we know from our service definition is a projection onto the corresponding entity in the remote service definition:

curl --url 'localhost:4004/odata/v4/main/Products?$top=1'

we get a successful result:

{
  "@odata.context": "$metadata#Products",
  "value": [
    {
      "ProductID": 1,
      "ProductName": "Chai",
      "QuantityPerUnit": "10 boxes x 20 bags",
      "UnitPrice": 18,
      "Category_CategoryID": 1,
      "Supplier_SupplierID": 1,
      "UnitsInStock": 39,
      "UnitsOnOrder": 0,
      "ReorderLevel": 10,
      "Discontinued": false
    }
  ]
}

This is due to the in-process based connectivity available in the single CAP server process.

Let's stop the CAP server at this point.

Mock the remote service in a separate process

In a second terminal window, let's now start the standalone mocking of the required northbreeze service with the cds mock command, like this:

cds mock northbreeze

We should see output like this:

[cds] - using bindings from: { registry: '~/.cds-services.json' }
[cds] - connect to db > sqlite { database: ':memory:' }
  > init from db/data/northbreeze.Products.json
  > init from db/data/northbreeze.Categories.csv
/> successfully deployed to in-memory database.

[cds] - mocking northbreeze {
  at: [ '/odata/v4/northbreeze' ],
  decl: 'srv/external/northbreeze.csn:170'
}
[cds] - server listening on { url: 'http://localhost:42623' }

The initial data is loaded as before, the northbreeze service is served, but crucially:

the provided Main service is not served (as we haven't asked it to be)
the required northbreeze service is available on a non-standard (in fact random) port 42623

We can successfully request resources in this mocked remote service at http://localhost:42623/odata/v4/northbreeze.

Start a normal CAP server process to have the local service served

Now in the first terminal window, let's restart the CAP server with cds watch, and we should now see:

[cds] - using bindings from: { registry: '~/.cds-services.json' }
[cds] - connect to db > sqlite { url: ':memory:' }
/> successfully deployed to in-memory database.

[cds] - serving Main {
  at: [ '/odata/v4/main' ],
  decl: 'srv/main.cds:3'
}
[cds] - server listening on { url: 'http://localhost:4004' }

Now, there's no initial data loaded, because that belongs to the required remote northbreeze service and thus not relevant here, because only the Main service is being served.

Why is only the provided Main service being served, and not the required remote northbreeze service, like before?

Get to know the binding registry

To answer that question, we need to recall that highlighted part of the help for the mocking option to cds serve earlier:

It starts in-process mock services for all required services configured in package.json#cds.requires, which don't have external bindings in the current process environment.

Here, in our local-first development context, our "current process environment" is effectively any (and all) CAP server process(es) running locally.

You might have noticed this line appearing in previous CAP server output samples in this post:

[cds] - using bindings from: { registry: '~/.cds-services.json' }

As they start up and shut down, local CAP server processes read and write to this registry file ~/.cds-services.json. They read it to see what services are available (that they might be requiring), and write to it to record the services they're providing (for other locally running CAP server processes).

When we started the separate CAP server to mock the northbreeze remote service with cds mock northbreeze, information was written to this file, recording the fact that this northbreeze service is "provided":

{
  "cds": {
    "provides": {
      "northbreeze": {
        "endpoints": {
          "odata": "/odata/v4/northbreeze"
        },
        "server": 17250
      }
    },
    "servers": {
      "17250": {
        "root": "file:///work/gh/github.com/qmacro/cap-nodejs-local-first-development/remoteservice",
        "url": "http://localhost:42623"
      }
    }
  }
}

As we can see, it also records where the provision is, in this case at http://localhost:42623, which is at the port that the mock server is listening on.

This is why a random port is not such a problem here.

Retry the local to remote proxy - part 1

Now that we have two CAP server processes running, one mocking the required remote service northbreeze, and the other serving the local service Main, let's retry that same request (to get the product data from the mocked remote service, proxied through the local service definition):

curl --url 'localhost:4004/odata/v4/main/Products?$top=1'

Oh dear:

{
  "error": {
    "message": "Entity \"Main.Products\" is annotated with \"@cds.persistence.skip\" and cannot be served generically.",
    "code": "501",
    "@Common.numericSeverity": 4
  }
}

The in-process connectivity available (provided for convenience) cannot be used here, and we have to implement some basic query and connectivity logic which is exactly what we will have to do in a productive scenario anyway.

So let's do that, using the simplest thing that could possibly work - adding this to a corresponding srv/main.js file:

const cds = require('@sap/cds')
module.exports = cds.service.impl(async function() {
  const northbreeze = await cds.connect.to('northbreeze')
  this.on('READ', 'Products', async (req) => {
    return await northbreeze.run(req.query)
  })
})

When a query arrives to read Products data, send it across to northbreeze and pass back whatever is returned.

Once the CAP server in the first terminal window restarts because of this change, note that there's a new log line in the output:

[cds] - connect to northbreeze > odata { url: 'http://localhost:42623/odata/v4/northbreeze' }

This is a direct result of the await cds.connect.to('northbreeze') line above.

Retry the local to remote proxy - part 2

With this simple implementation in place, retrying that same request again will result in some very satisfying log output in both CAP server processes.

Before we do, stop the CAP server in the first terminal window and restart it specifying DEBUG=remote like this, to get more log output detail for remote related activities:

DEBUG=remote cds watch

Now, after retrying the request for a final time, we see this log output in the first (Main) CAP server log output:

[remote] - GET http://localhost:42623/odata/v4/northbreeze/Products?$select=ProductID,ProductName,QuantityPerUnit,UnitPrice,Category_CategoryID,Supplier_SupplierID,UnitsInStock,UnitsOnOrder,ReorderLevel,Discontinued&$top=1 {
  headers: {
    accept: 'application/json,text/plain',
    'x-correlation-id': 'ff4da315-ec6e-4e89-97d4-85f96b270ed6',
    'x-correlationid': 'ff4da315-ec6e-4e89-97d4-85f96b270ed6'
  }
}
[remote] - Executing via native fetch.

We can see that there's an HTTP request, specifically an OData QUERY operation, that's been constructed and sent to the URL where the remote service is being provided³.

And in the CAP server process in the second terminal window, where we're separately mocking the northbreeze remote service on 42623, we see that OData QUERY operation arrive:

[odata] - GET /odata/v4/northbreeze/Products {
  '$select': 'ProductID,ProductName,QuantityPerUnit,UnitPrice,Category_CategoryID,Supplier_SupplierID,UnitsInStock,UnitsOnOrder,ReorderLevel,Discontinued',
  '$top': '1'
}

A true inter-process remote service call. All running locally, and orchestrated in the simplest way possible.

Wrapping up

This post has just scratched the surface of what's possible when it comes to working in local-first development mode, with remote services. For more information and stuff that you can practise yourself, we have the CAP Service Integration CodeJam exercises publicly available for you. Happy learning!

Footnotes

"odd" is short for OData Deep Dive, and the service is used in the corresponding SAP Tutorial Navigator mission that I'm rewriting currently - see OData Deep Dive rewrite in the open for details.
Note that since the CAP Apr 2026 release, the SAP Cloud SDK is no longer mandatory for remote communication in development scenarios.
We can also see, via the "Executing via native fetch" line, that the Node-native Fetch API is in use here now, since Apr 2026.

Local-first dev with CAP Node.js

2026-05-11T00:00:00Z

This series post is related to a talk I'm putting together:

Local-first development with CAP Node.js - mock all the things!

As developers we need to be free of distractions plus a tight and speedy development loop. But definitely not at the expense of ignoring or postponing important design decisions. CAP's mocking facilities abstracts us from much tedium and ceremony, allowing us to iterate fast on data, auth, messaging and remote services while we develop. This session shows you what, and how.

There's a repo for the themes and demos in this talk at https://github.com/qmacro/cap-nodejs-local-first-development/.

You can read more about the context of this talk in the section below.

The posts

Mocking auth (11 May)
Mocking remote services (13 May)
Mocking data (coming soon)
Mocking messaging (coming soon)

More context

There are a couple of aspects of development that come together for the perfect (positive) storm of focused rapid iteration that results in a solid and complete foundation for a production offering, from the outset. They are:

a tight feedback loop within which we can iterate rapidly on design and implementation
local-first facilities for everything we need to get going, with minimum setup and configuration

As part of this second aspect, being able to easily fold in key design requirements and real world facilities from the very start means that we don't avoid them, or put them off until it's too late. Instead, we can embrace and address them right from the start of the iteration cycles, and avoid the build-up of design debt. The CAP development kit includes tools and affordances that make this easy for us to do, in the form of mocking.

Local-first dev with CAP Node.js - mocking auth

2026-05-11T00:00:00Z

This post is one of a series on local-first development with CAP Node.js.

The mocked authentication strategy

The Authentication Strategies section of the Node.js Security topic in Capire explains the different strategies available, and the "mocked" strategy comes with pre-defined users that can be used, with their various levels of authorisations, to explore, define and test security-related constructs. This mock user configuration can be modified and extended too, but what comes out of the box is definitely enough to get started.

Working through an example

In this post, we'll work through an example of mocking auth, based on content in the auth/ directory of the talk repository.

Note that what's absent here is any form of auth implementation - all declarations available are automatically enforced by CAP's generic service providers.

The service definition

In srv/main.cds there's a single service defined, with a couple of entities that are simple projections on to the entities in the data model:

using northwind from '../db/schema';

service Main {

  entity Products   as projection on northwind.Products;
  entity Categories as projection on northwind.Categories;

}

Starting a CAP server in local development mode with cds watch shows us that the mocked authentication strategy is in play by default:

[cds] - using auth strategy { kind: 'mocked' }
[cds] - serving Main {
  at: [ '/main' ],
  decl: 'srv/main.cds:4'
}

As we haven't yet addressed any auth requirements in our CDS model, access is currently open to all, as we can see¹:

; curl \
  --include \
  --url 'localhost:4004/main/Products?$top=1'
HTTP/1.1 200 OK
OData-Version: 4.0
Content-Type: application/json; charset=utf-8
Content-Length: 105

{
  "@odata.context": "$metadata#BasicProducts",
  "value": [
    {
      "ID": 1,
      "name": "Chai",
      "supplier": "Exotic Liquids"
    }
  ]
}

Examine the pre-defined users and their authorisations

We can take a look at the pre-defined user data that is defined for the mocked authentication strategy, with:

cds env requires.auth.users

which will emit something like this:

{
  alice: { tenant: 't1', roles: [ 'admin' ] },
  bob: { tenant: 't1', roles: [ 'cds.ExtensionDeveloper' ] },
  carol: { tenant: 't1', roles: [ 'admin', 'cds.ExtensionDeveloper' ] },
  dave: { tenant: 't1', roles: [ 'admin' ], features: [] },
  erin: { tenant: 't2', roles: [ 'admin', 'cds.ExtensionDeveloper' ] },
  fred: { tenant: 't2', features: [ 'isbn' ] },
  me: { tenant: 't1', features: [ '*' ] },
  yves: { roles: [ 'internal-user' ] },
  '*': true
}

Restrict the service

Let's annotate the service with some basic role based access control (RBAC) requirements - that of needing to authenticate, via the pseudo-role authenticated-user. We can use the @requires annotation:

using northwind from '../db/schema';

@requires: 'authenticated-user'
service Main {

  ...

}

The same curl request as before now fails with an appropriate HTTP 401 status code:

HTTP/1.1 401 Unauthorized
WWW-Authenticate: Basic realm="Users"
Content-Type: text/plain; charset=utf-8
Content-Length: 12

Unauthorized

Authenticate with a pre-defined user

We can re-try the request with one of the pre-defined users²; because the requirement is just for the pseudo-role authenticated-user, we don't need any particular actual role allocated to the user, we just need to be successfully authenticated (and so identified) in this case:

; curl \
  --user alice: \
  --include \
  --url 'localhost:4004/main/Products?$top=1'
HTTP/1.1 200 OK
OData-Version: 4.0
Content-Type: application/json; charset=utf-8
Content-Length: 105

{
  "@odata.context": "$metadata#BasicProducts",
  "value": [
    {
      "ID": 1,
      "name": "Chai",
      "supplier": "Exotic Liquids"
    }
  ]
}

Try some finer-grained access restrictions

With the @restrict we can define finer grained access requirements³ in privilege building blocks of this form:

{ grant:<events>, to:<roles>, where:<filter-condition> }

Let's now add privilege requirements for the Categories entity, like this:

using northwind from '../db/schema';

@requires: 'authenticated-user'
service Main {

  entity Products   as projection on northwind.Products;

  @restrict: [
    {
      grant: 'WRITE',
      to   : 'buyer'
    },
    {
      grant: 'READ',
      to   : 'any'
    }
  ]
  entity Categories as projection on northwind.Categories;
}

This says that any (authenticated) user can read the categories, but only a user with the buyer role can perform "write"-semantic operations.

Confirm read operations are permitted

Let's check that "read"-semantic operations are allowed for authenticated users (remember that the entity access is also governed by the authenticated-user pseudo-role restriction on the service that contains it):

; curl \
  --user alice: \
  --include \
  --url 'localhost:4004/main/Categories?$top=1'
HTTP/1.1 200 OK
OData-Version: 4.0
Content-Type: application/json; charset=utf-8
Content-Length: 155

{
  "@odata.context": "$metadata#Categories",
  "value": [
    {
      "CategoryID": 1,
      "CategoryName": "Beverages",
      "Description": "Soft drinks, coffees, teas, beers, and ales"
    }
  ]
}

Looks OK.

Try a write operation

Now for a "write"-semantic operation. Let's go big and try DELETE:

; curl \
  --user alice: \
  --include \
  --request DELETE \
  --url 'localhost:4004/main/Categories/1'
HTTP/1.1 403 Forbidden
OData-Version: 4.0
Content-Type: application/json; charset=utf-8
Content-Length: 74

{
  "error": {
    "message": "Forbidden",
    "code": "403",
    "@Common.numericSeverity": 4
  }
}

Alice, with the admin role, is denied.

Add a user and role to for the mocked strategy

We can also modify and add to the pre-defined user definitions for the mocked authentication strategy. Let's do that, adding a couple of extra roles for Alice in a separate .cdsrc.json file in the project:

{
  "cds": {
    "requires": {
      "auth": {
        "users": {
          "alice": {
            "roles": [
              "admin",
              "buyer",
              "head-office"
            ]
          }
        }
      }
    }
  }
}

One of the roles is buyer, so let's now retry the previous request:

; curl \
  --user alice: \
  --include \
  --request DELETE \
  --url 'localhost:4004/main/Categories/1'
HTTP/1.1 204 No Content
OData-Version: 4.0

Success!

Wrapping up

With the mocked authentication strategy, we can embrace and work on the important aspect of securing our app or service right from the very start. CAP makes it easy to do the right things here.

For more information, see the Authentication topic in Capire.

Footnotes

The JSON output in these examples has been pretty-printed for readability here.
The --user option for curl allows us to specify a username and password separated by a colon, so alice: here is just the username combined with an empty password (there are no passwords for these users). If we'd just specified --user alice without a colon, then curl would have prompted us for a password - we could have then just pressed Enter but this is one step we can avoid.
In fact, @requires is just a convenience shortcut for @restrict. The annotation
```
@requires: 'authenticated-user'
```
that we used earlier is equivalent to
```
@restrict: [ { grant: '*', to: 'authenticated-user' } ]
```

CDS expressions in CAP - notes on Part 6

2026-05-04T00:00:00Z

See the series post for an overview of all the episodes.

Introduction

00:00 Introduction and recap.

05:47 Patrice takes over and looks at the syntax diagram in the CXL topic of Capire and the specific path expression diagram. He remarks that one of the cool things about path expressions is that you can chain navigations together, as is shown in one of the examples that follow the diagram:

assoc[filter].struct.assoc.element

Such navigation paths are materialised at some point (on use) into constructs such as:

an EXISTS subquery
a LEFT JOIN
a correlated subquery (for the expands)

These path expressions in particular (as well as CXL in general) can be used everywhere; Patrice gives examples:

in CDL models (when defining the schema or service projections)
in annotation expressions (when referring to a path or element)
in queries (written in CQL)

10:26 As good things come in threes, or so they say, here's a final three-bullet point list (making a total of three lists, too), where Patrice enumerates the different contexts in which we used the nonSeller association-like calculated element (nonSeller = books[ stock > 170 ]), in the previous episode:

12:29 We get a glimpse behind the scenes at the @cap-js/db-service mechanisms, where Patrice shows the function that is used to generate aliases, in the context of the new (to us in this series) style of aliases shown in the normalised, intermediate CAP-style SQL:

> cds.ql`
  SELECT from ${Authors}
  {name}
  where exists nonSeller
  `.forSQL()
cds.ql {
  SELECT: {
    from: { ref: [ 'sap.capire.bookshop.Authors' ], as: '$A' },
    columns: [ { ref: [ '$A', 'name' ] } ],
    where: [
      'exists',
      {
        SELECT: {
          from: { ref: [ 'sap.capire.bookshop.Books' ], as: '$n' },
          columns: [ { val: 1 } ],
          where: [
            {
              xpr: [
                { ref: [ '$n', 'author_ID' ] },
                '=',
                { ref: [ '$A', 'ID' ] }
              ]
            },
            'and',
            { xpr: [ { ref: [ '$n', 'stock' ] }, '>', { val: 170 } ] }
          ]
        }
      }
    ]
  }
}

These $-prefixed short alias names are "technical" aliases. There's a function that Patrice dives into, specifically getImplicitAlias in @cap-js/db-service/lib/utils.js, that has a useTechnicalAlias parameter which defaults to true.

At this point we'll make the transition to having these technical aliases shown in our CAP-style SQL, as shown in this example, instead of the human-centric ones we've had so far.

Looking at the genres entity definition

16:02 Patrice takes some time to explain the reason for the technical aliases, using the recursively structured Genres entity definition as an example:

entity Genres : sap.common.CodeList {
  key ID       : Integer;
      parent   : Association to Genres;
      children : Composition of many Genres
                   on children.parent = $self;
}

Querying the genre information in Patrice's sample project we see the general idea:

> await cds.ql`
  SELECT from ${Genres}
  { name, parent { * } }
  `
[
  { name: 'Fiction', parent: null },
  { name: 'Drama', parent: { name: 'Fiction', descr: null, ID: 10, parent_ID: null } },
  { name: 'Poetry', parent: { name: 'Fiction', descr: null, ID: 10, parent_ID: null } },
  { name: 'Fantasy', parent: { name: 'Fiction', descr: null, ID: 10, parent_ID: null } },
  { name: 'Science Fiction', parent: { name: 'Fiction', descr: null, ID: 10, parent_ID: null } },
  { name: 'Romance', parent: { name: 'Fiction', descr: null, ID: 10, parent_ID: null } },
  { name: 'Mystery', parent: { name: 'Fiction', descr: null, ID: 10, parent_ID: null } },
  { name: 'Thriller', parent: { name: 'Fiction', descr: null, ID: 10, parent_ID: null } },
  { name: 'Dystopia', parent: { name: 'Fiction', descr: null, ID: 10, parent_ID: null } },
  { name: 'Fairy Tale', parent: { name: 'Fiction', descr: null, ID: 10, parent_ID: null } },
  { name: 'Non-Fiction', parent: null },
  { name: 'Biography', parent: { name: 'Non-Fiction', descr: null, ID: 20, parent_ID: null } },
  { name: 'Autobiography', parent: { name: 'Biography', descr: null, ID: 21, parent_ID: 20 } },
  { name: 'Essay', parent: { name: 'Non-Fiction', descr: null, ID: 20, parent_ID: null } },
  { name: 'Speech', parent: { name: 'Non-Fiction', descr: null, ID: 20, parent_ID: null } }
]

To help visualise the genre hierarchy, here it is represented in a tree structure:

/tmp/genres/
├── Fiction (10)
│   ├── Drama
│   ├── Dystopia
│   ├── Fairy Tale
│   ├── Fantasy
│   ├── Mystery
│   ├── Poetry
│   ├── Romance
│   ├── Science Fiction
│   └── Thriller
└── Non-Fiction (20)
    ├── Biography (21)
    │   └── Autobiography
    ├── Essay
    └── Speech

Because we'll be referring to them later in this post, the IDs for Fiction, Non-Fiction and Biography (genres that are "parents") are shown in brackets.

An introduction to scoped queries

16:44 At this point Patrice introduces us to "scoped queries", where we can traverse the FROM with a path expression like construct:

> await cds.ql`
  SELECT from ${Genres}:parent
  { name }
  `
[
  { name: 'Fiction' },
  { name: 'Non-Fiction' },
  { name: 'Biography' }
]

Here's another "wait, what's this and what's going on?!" moment in this series :-)

Let's have a look at the CQN:

> q = cds.ql`
  SELECT from ${Genres}:parent
  { name }
  `
cds.ql {
  SELECT: {
    from: { ref: [ 'sap.capire.bookshop.Genres', 'parent' ] },
    columns: [ { ref: [ 'name' ] } ]
  }
}

The value for from looks very much like a path expression - a ref with a value that is an array of multiple elements [ 'sap.capire.bookshop.Books', 'parent' ].

Next, let's have a look at the CAP-style SQL:

> q.forSQL()
cds.ql {
  SELECT: {
    from: { ref: [ 'sap.capire.bookshop.Genres' ], as: '$p' },
    columns: [ { ref: [ '$p', 'name' ] } ],
    where: [
      'exists',
      {
        SELECT: {
          from: { ref: [ 'sap.capire.bookshop.Genres' ], as: '$G' },
          columns: [ { val: 1 } ],
          where: [
            { ref: [ '$G', 'parent_ID' ] },
            '=',
            { ref: [ '$p', 'ID' ] }
          ]
        }
      }
    ]
  }
}

Taking a moment to stare at this the mist starts to clear. Looking at this intermediate form, we see:

there's an EXISTS clause that's been materialised
the "lead" and "related" entities involved are the same, here (Genres)
the main query retrieves the genre name
the subquery returns something based on a condition that correlates a relationship within this single entity
that condition is that the ID of the genre whose name we're retrieving in the main query ... matches a parent ID value, at least once

In other words, "retrieve the genres that are parents" (i.e. that have children).

This can be stretched out to another path level, to retrieve genres that are "grandparents" (i.e. that are parents of parents):

> q = cds.ql`
  SELECT from ${Genres}:parent.parent
  { name }
  `
cds.ql {
  SELECT: {
    from: { ref: [ 'sap.capire.bookshop.Genres', 'parent', 'parent' ] },
    columns: [ { ref: [ 'name' ] } ]
  }
}

This returns just a single result in the set:

> await q
[
  { name: 'Non-Fiction' }
]

This makes sense, as we can see in the tree structure earlier, Non-Fiction is the only genre that is a parent of a parent:

/tmp/genres/
...
└── Non-Fiction (20)
    ├── Biography (21)
    │   └── Autobiography
    ├── Essay
    └── Speech

Further examples of scoped queries

I initially struggled with this example. Illustrative as it was (especially with the parent.parent part), it was one that was a little complicated due to its self-referential nature.

So I used the power & utility of the cds REPL to explore more, and things started to make sense. For example, this query gives us the genres for books written by authors 40 years old or younger (Carpenter and the two Brontë sisters):

> q = cds.ql`
  SELECT from Authors[age <= 40]:books.genre
  { name }
  `
cds.ql {
  SELECT: {
    from: {
      ref: [
        {
          id: 'Authors',
          where: [ { ref: [ 'age' ] }, '<=', { val: 40 } ]
        },
        'books',
        'genre'
      ]
    },
    columns: [ { ref: [ 'name' ] } ]
  }
}
> await q
[
  { name: 'Drama' },
  { name: 'Romance' },
  { name: 'Mystery' }
]

Again we have a path expression traversal happening in the from part of the query, this time with an infix filter on the first part of the path. I used an infix filter here to make the example a little more interesting.

I then wondered to myself what it would look and feel like to use an infix filter on a different part of the path, and came up with this:

> q = cds.ql`
  SELECT from ${Authors}:books[title like 'The %'].genre
  { name }
  `
cds.ql {
  SELECT: {
    from: {
      ref: [
        'sap.capire.bookshop.Authors',
        {
          id: 'books',
          where: [ { ref: [ 'title' ] }, 'like', { val: 'The %' } ]
        },
        'genre'
      ]
    },
    columns: [ { ref: [ 'name' ] } ]
  }
}

This produces the names of the genres of books that begin with the definite article ("The ..."):

> q = cds.ql`
  SELECT from ${Authors}:books[title like 'The %'].genre
  { name }
  `
cds.ql {
  SELECT: {
    from: {
      ref: [
        'sap.capire.bookshop.Authors',
        {
          id: 'books',
          where: [ { ref: [ 'title' ] }, 'like', { val: 'The %' } ]
        },
        'genre'
      ]
    },
    columns: [ { ref: [ 'name' ] } ]
  }
}
> await q
[
  { name: 'Fantasy' },
  { name: 'Mystery' }
]

At first I thought the first part of the path in this query would now be redundant, assuming that I could have just written:

> q = cds.ql`
  SELECT from ${Books}[title like 'The %'].genre
  { name }
  `

But I was mistaken.

The key to thinking about this is remembering the term "scoped". To illustrate, let's adjust the postfix projection to have the book titles returned, and run the scoped query:

> await cds.ql`
  SELECT from ${Authors}:books[title like 'The %']
  { title }
  `
[
  { title: 'The Raven' },
  ... (12 rows removed for brevity)
  { title: 'The Fall of Gondolin' }
]

Running the unscoped query returns the same result set, yes:

> await cds.ql`
  SELECT from ${Books}[title like 'The %']
  { title }
  `
[
  { title: 'The Raven' },
  ... (12 rows removed for brevity)
  { title: 'The Fall of Gondolin' }
]

At this point let's add a book, but without a connection to an author, like this:

> await INSERT.into(Books,[{ID:999,title: 'The Book With No Author!'}])
InsertResult { results: [ { changes: 1, lastInsertRowid: 999 } ] }

Now re-running those queries gives us different result sets. First, the new book is included in the result set for the unscoped query:

> await cds.ql`
  SELECT from ${Books}[title like 'The %']
  { title }
  `
[
  { title: 'The Raven' },
  ... (12 rows removed for brevity)
  { title: 'The Fall of Gondolin' },
  { title: 'The Book With No Author!' }
]

But it is not included in the result set for the scoped query:

> await cds.ql`
  SELECT from ${Authors}:books[title like 'The %']
  { title }
  `
[
  { title: 'The Raven' },
  ... (12 rows removed for brevity)
  { title: 'The Fall of Gondolin' }
]

While both queries are materialised into SELECTs on sap.capire.bookshop.Books, the scoped query constrains the result set to those entries where there is an author, as we can see with the EXISTS in the intermediate SQL:

> cds.ql`
  SELECT from ${Authors}:books[title like 'The %']
  { title }
  `.forSQL()
cds.ql {
  SELECT: {
    from: { ref: [ 'sap.capire.bookshop.Books' ], as: '$b' },
    columns: [ { ref: [ '$b', 'title' ] } ],
    where: [
      'exists',
      {
        SELECT: {
          from: { ref: [ 'sap.capire.bookshop.Authors' ], as: '$A' },
          columns: [ { val: 1 } ],
          where: [
            { ref: [ '$A', 'ID' ] },
            '=',
            { ref: [ '$b', 'author_ID' ] }
          ]
        }
      },
      'and',
      { ref: [ '$b', 'title' ] },
      'like',
      { val: 'The %' }
    ]
  }
}

See the section Combining the scoped syntax with infix filters later on in these notes for more on this.

Back to looking at the technical aliases

At 18:40 Patrice steers us back to thinking about technical aliases by adding an explicit alias to the path expression in the query:

> q = cds.ql`
  SELECT from ${Genres}:parent as parent
  { name, parent.ID }
  `
cds.ql {
  SELECT: {
    from: { ref: [ 'sap.capire.bookshop.Genres', 'parent' ], as: 'parent' },
    columns: [ { ref: [ 'name' ] }, { ref: [ 'parent', 'ID' ] } ]
  }
}

Let's just see what this emits when executed:

> await q
[
  { name: 'Fiction', ID: 10 },
  { name: 'Non-Fiction', ID: 20 },
  { name: 'Biography', ID: 21 }
]

The alias is to the path expression in the query, and is an explicit one. It was also used explicitly in the parent.ID construct. There's also an implicit alias in that the referenced name element is really parent.name.

But things get confusing if we want to refer to elements in the parent genre's parent, as Patrice demonstrates:

> q = cds.ql`
  SELECT from ${Genres}:parent as parent
  { name, parent.parent.ID }
  `
cds.ql {
  SELECT: {
    from: { ref: [ 'sap.capire.bookshop.Genres', 'parent' ], as: 'parent' },
    columns: [ { ref: [ 'name' ] }, { ref: [ 'parent', 'parent', 'ID' ] } ]
  }
}

Note the three level path expression:

{ ref: [ 'parent', 'parent', 'ID' ] }

This returns:

> await q
[
  { name: 'Fiction', parent_ID: null },
  { name: 'Non-Fiction', parent_ID: null },
  { name: 'Biography', parent_ID: 20 }
]

which is what we should expect, in that Fiction and Non-Fiction are top level genres and have no parents.

Clearly there's a high degree of potential confusion and conflict, and so technical aliases make more sense in the runtime (i.e. implicitly) as they have far less chance of clashing.

At this point, it wouldn't be a bad idea to go and get a coffee and then come back for the rest of this write-up :-)

From nested expands to dot notation projections

23:02 At this point we revisit a nested expand from last week:

> await cds.ql`
  SELECT from ${Authors}
  { name, nonSeller { title, stock } }
  where exists nonSeller
  `
[
  {
    name: 'Richard Carpenter',
    nonSeller: [ { title: 'Catweazle', stock: 187 } ]
  },
  {
    name: 'J. R. R. Tolkien',
    nonSeller: [
      { title: 'Unfinished Tales', stock: 189 },
      { title: 'The Children of Húrin', stock: 203 },
      { title: 'Beren and Lúthien', stock: 178 },
      { title: 'The Fall of Gondolin', stock: 195 }
    ]
  }
]

Patrice reminds us that such an expand is "just another postfix projection", which leads to the possibility of using * and excluding clauses, as we saw in the previous episode on this topic:

{ name, nonSeller { * } excluding { ID } }

With such expands being "variants" of postfix projections, we are then introduced to another variant, which looks similar but does something different.

To illustrate, we move up to the service layer and go to the Books projection in the CatalogService, which currently looks like this, which includes a single element (the author name) from the author association:

entity Books       as
  projection on my.Books {
    *,
    author.name as author
  }
  excluding {
    createdBy,
    modifiedBy
  };

If we wanted to add another element from the author association, we could do this:

entity Books       as
  projection on my.Books {
    *,
    author.name as author,
    author.dateOfBirth
  }
  excluding {
    createdBy,
    modifiedBy
  };

At 26:27 Patrice shows what this resolves to at the database layer, by running a build for HANA:

cds build --profile production

This produces the HANA artifacts for deploying via the HDI container. One of these artifacts is the gen/db/src/gen/CatalogService.Books.hdbview file which contains the DDL statement to create the view that represents this projection:

VIEW CatalogService_Books AS SELECT
  Books_0.createdAt,
  Books_0.modifiedAt,
  Books_0.title,
  Books_0.ID,
  Books_0.descr,
  author_1.name AS author,
  Books_0.genre_ID,
  Books_0.stock,
  Books_0.price,
  Books_0.currency_code,
  author_1.dateOfBirth
FROM (
  sap_capire_bookshop_Books AS Books_0
  LEFT JOIN sap_capire_bookshop_Authors AS author_1
    ON Books_0.author_ID = author_1.ID
)

Doing this for the CDL is the rough equivalent to when we've been examining the corresponding SQL for our queries in CQL, using toSQL() in the cds REPL.

Here's what Patrice had to say about this DDL:

technical aliases (Books_0 and author_1) were used by the compiler, minimising ambiguities
both name and dateOfBirth use the same JOIN node

Now we have defined this and have confirmed at the DDL level what we expect to see, we can now explore the dot notation, which is essentially a little bit of syntactic sugar that is arguably easier on the eye:

entity Books       as
  projection on my.Books {
    *,
    author.{
      name,
      dateOfBirth
    }
  }
  excluding {
    createdBy,
    modifiedBy
  };

We can easily think of this as opening another projection (on the author association) because of the use of braces, but be aware that this is not a nested expand, it is a shortcut to, or a summarised version of, multiple path expressions with the same root.

A look at the compiled DDL for this variant shows that it's pretty much the same, i.e. still a flat list of elements, essentially:

VIEW CatalogService_Books AS SELECT
  Books_0.createdAt,
  Books_0.modifiedAt,
  Books_0.title,
  Books_0.ID,
  Books_0.descr,
  Books_0.author_ID,
  Books_0.genre_ID,
  Books_0.stock,
  Books_0.price,
  Books_0.currency_code,
  author_1.name AS name,
  author_1.dateOfBirth AS dateOfBirth
FROM (
  sap_capire_bookshop_Books AS Books_0
  LEFT JOIN sap_capire_bookshop_Authors AS author_1
    ON Books_0.author_ID = author_1.ID
)

We do get the Books_0.author_ID element additionally here, but that is of little consequence.

The power of infix filters

Patrice goes one step further at 30:45 to show how an infix filter might be added to this dot notation construct; the filter chosen isn't allowed in this context, but something like this is:

entity Books       as
  projection on my.Books {
    *,
    author[isAlive].{
      name as thename,
      dateOfBirth
    }
  }
  excluding {
    createdBy,
    modifiedBy
  };

In fact, Patrice uses this condition a little bit later on, albeit in a slightly longer form isAlive = true.

There is a wider range of possibilities here in the context of queries, which Patrice then demonstrates in the cds REPL with:

> await cds.ql`
  SELECT from ${Books}
  {
    title as book,
    author[exists books[genre.name = 'Fantasy']].{ name, age }
  }
  `
[
  { book: 'Wuthering Heights', author_name: null, author_age: null },
  { book: 'Jane Eyre', author_name: null, author_age: null },
  { book: 'The Raven', author_name: null, author_age: null },
  { book: 'Eleonora', author_name: null, author_age: null },
  {
    book: 'Catweazle',
    author_name: 'Richard Carpenter',
    author_age: 82
  },
  {
    book: 'Mistborn: The Final Empire',
    author_name: 'Brandon Sanderson',
    author_age: 50
  },
  {
    book: 'The Well of Ascension',
    author_name: 'Brandon Sanderson',
    author_age: 50
  },
  ...
]

Here we have a nested infix filter. Note that the condition construct only applies to the author data here, which explains why we have null values for some authors but not for others, within the books-led set.

Patrice highlights that one of the advantages of this dot notation approach, when used with infix filters, is that we can define our path conditions up front and then specify what we want when we follow that path, without having to repeat it.

This is also known as an inline nested projection.

Infix filter construction

At 36:36 Patrice shows us the syntax diagram that describes infix filters

and we talk a little about:

the where keyword, which is optional and often omitted, similar to other keywords in CDL such as define.
the main part which is an expression, which is thus essentially "anything"

This latter point means that, depending on the context used, infix filters will be materialised into different shapes.

Materialisation in DDL

To illustrate this somewhat, and to round this section out, at 38:18 Patrice modifies the infix filter condition in the CDL for the books projection to be dateOfBirth > 1900¹:

entity Books       as
  projection on my.Books {
    *,
    author[dateOfBirth > 1900].{
      name as thename,
      dateOfBirth
    }
  }
  excluding {
    createdBy,
    modifiedBy
  };

This has the effect that the DDL for the corresponding view in HANA is defined like this:

VIEW CatalogService_Books AS SELECT
  Books_0.createdAt,
  Books_0.modifiedAt,
  Books_0.title,
  Books_0.ID,
  Books_0.descr,
  Books_0.author_ID,
  Books_0.genre_ID,
  Books_0.stock,
  Books_0.price,
  Books_0.currency_code,
  author_1.name AS thename,
  author_1.dateOfBirth AS dateOfBirth
FROM (
  sap_capire_bookshop_Books AS Books_0
  LEFT JOIN sap_capire_bookshop_Authors AS author_1
    ON
      (Books_0.author_ID = author_1.ID)
      AND
      (author_1.dateOfBirth > 1900)
)

The infix filter is mixed in to the main foreign key matching part of the ON condition.

Infix filters in the FROM clause

Infix filters can also be used in the FROM clause in queries, as Patrice shows at 43:35:

> await cds.ql`
  SELECT from ${Books}[where stock between 50 and 100]
  { title, stock }
  `
[
  { title: 'Wuthering Heights', stock: 95 },
  { title: 'Jane Eyre', stock: 78 },
  { title: 'The Alloy of Law', stock: 67 },
  { title: 'Shadows of Self', stock: 89 },
  { title: 'Mistborn: Secret History', stock: 98 }
]

This is a nice syntactic sugar based variant, made even better by the use of the optional where, so we can read the entire construct (entity, infix filter condition and then the postfix projection) naturally, rather than e.g.:

> await cds.ql`
  SELECT from ${Books}
  { title, stock }
  where stock between 50 and 100
  `

These variants are the same, which we can see if we compare their intermediate SQL (via forSQL()):

cds.ql {
  SELECT: {
    from: { ref: [ 'sap.capire.bookshop.Books' ], as: '$B' },
    columns: [ { ref: [ '$B', 'title' ] }, { ref: [ '$B', 'stock' ] } ],
    where: [
      { ref: [ '$B', 'stock' ] },
      'between',
      { val: 50 },
      'and',
      { val: 100 }
    ]
  }
}

Combining the scoped syntax with infix filters

To drive home two of the key concepts we've looked at, Patrice now combines them, starting at 47:15. In other words, taking the scoped syntax variant of the FROM clause with a path expression constructed with a colon, and adding an infix filter.

To illustrate this and start simply, we first see this, which is just the scoped query part:

> q = cds.ql`SELECT from ${Books}:author { name }`
cds.ql {
  SELECT: {
    from: { ref: [ 'sap.capire.bookshop.Books', 'author' ] },
    columns: [ { ref: [ 'name' ] } ]
  }
}
> await q
[
  { name: 'Emily Brontë' },
  { name: 'Charlotte Brontë' },
  { name: 'Edgar Allen Poe' },
  { name: 'Richard Carpenter' },
  { name: 'Brandon Sanderson' },
  { name: 'J. R. R. Tolkien' }
]

This alone is worth dwelling on, when Patrice explains how he thinks about such constructs - reading from right to left: "select those authors for whom exist (at least one) book(s)". The selection is on the Authors entity, but it is constrained by the books-authors relationship.

If there was an author in the database for whom there were no corresponding book entries, this author would not be part of the result set. Rather than add an entry to show this, Patrice now adds on to this query an infix filter for the Books entity:

> q = cds.ql`
  SELECT from ${Books}[stock between 50 and 100]:author
  { name }
  `

Looking at the intermediate SQL, we can see that this stock based condition in the infix filter becomes part of the WHERE clause of the subquery used in the EXISTS:

> q.forSQL()
cds.ql {
  SELECT: {
    from: { ref: [ 'sap.capire.bookshop.Authors' ], as: '$a' },
    columns: [ { ref: [ '$a', 'name' ] } ],
    where: [
      'exists',
      {
        SELECT: {
          from: { ref: [ 'sap.capire.bookshop.Books' ], as: '$B' },
          columns: [ { val: 1 } ],
          where: [
            { ref: [ '$B', 'author_ID' ] },
            '=',
            { ref: [ '$a', 'ID' ] },
            'and',
            { ref: [ '$B', 'stock' ] },
            'between',
            { val: 50 },
            'and',
            { val: 100 }
          ]
        }
      }
    ]
  }
}

This has the expected effect - a reduced result set:

> await q
[
  { name: 'Emily Brontë' },
  { name: 'Charlotte Brontë' },
  { name: 'Brandon Sanderson' }
]

And the possibilities don't end there, of course, which Patrice shows at this point by gratuitously adding another infix clause thus:

> await cds.ql`
  SELECT from ${Books}[stock between 50 and 100]:author[order by name asc]
  { name }
  `
[
  { name: 'Brandon Sanderson' },
  { name: 'Charlotte Brontë' },
  { name: 'Emily Brontë' }
]

At this point in this writeup, I wonder about the contents of this second infix clause, in that it's not really a filter in the simple way I understand filters so I wanted to try to add something that feels more like an actual "restrictive" filter², and that works too:

> await cds.ql`
  SELECT from ${Books}[stock between 50 and 100]:author[name like '%Brontë']
  { name }
  `
[
  { name: 'Emily Brontë' },
  { name: 'Charlotte Brontë' }
]

Disambiguating elements and their scoped named containers with colons

At 50:00 I ask Patrice to explain a little bit more about the use of the colon (:) when constructing a "fully qualified" element reference. Normally we would use a dot to express traversal through a relationship, such as author.name. But when it comes to including the "container" name, most commonly an entity, we must be precise and unambiguous, especially in the context of how the compiler will interpret what we express.

With the use of namespaces, defined with either the namespace or context directives, or even just expressed explicitly (e.g. entity Foo.Bar { ... }), a question arises: "Where does the name of the container end and the name of the element (path) start?". Sometimes, a colon is needed to facilitate the answer to that question.

Here's an example. Consider this simple model:

namespace a;

context b.c {

  entity D {
    e : String;
  }

}

If we wanted to annotate the element e with @readonly³, we might write:

annotate a.b.c.D.e with @readonly;

However, the compiler would emit a warning:

Artifact “a.b.c.D.e” has not been found

To disambiguate where the "join" is, we use a colon⁴:

annotate a.b.c.D : e with @readonly;

Summarising infix filters

At 52:58 Patrice gives a nice summary concerning infix filters, which can be employed in many places - everywhere that you can use paths, effectively - and are materialised differently depending on where and how they're used:

in a LEFT JOIN
in an EXISTS subquery
in a correlated subquery

Yes, it's not a coincidence that this is the same list that we considered at the start of this episode.

Here follows examples of each of those.

In a service projection (a JOIN example)

One of these places we saw was in the Books projection in the CatalogService (see the Materialisation in DDL section):

author[dateOfBirth > 1900].name as authorName

This infix filter is materialised in the JOIN

VIEW CatalogService_Books AS SELECT
  ...
  Books_0.author_ID,
  ...
  author_1.name AS authorName,
  author_1.dateOfBirth AS dateOfBirth
FROM (
  sap_capire_bookshop_Books AS Books_0
  LEFT JOIN sap_capire_bookshop_Authors AS author_1
    ON (Books_0.author_ID = author_1.ID)
    AND (author_1.dateOfBirth > 1900)
)

In a scoped query (an EXISTS example)

The infix filter in the scoped query we saw earlier (see the Combining the scoped syntax with infix filters section):

> q = cds.ql`
  SELECT from ${Books}[stock between 50 and 100]:author
  { name }
  `

is pushed down and added to the EXISTS subquery when the query is materialised; we can see this nicely even in the intermediate (CAP-style) SQL:

cds.ql {
  SELECT: {
    from: { ref: [ 'sap.capire.bookshop.Authors' ], as: '$a' },
    columns: [ { ref: [ '$a', 'name' ] } ],
    expand: 'root',
    orderBy: [ { ref: [ 'name' ], sort: 'asc' } ],
    where: [
      'exists',
      {
        SELECT: {
          from: { ref: [ 'sap.capire.bookshop.Books' ], as: '$B' },
          columns: [ { val: 1 } ],
          where: [
            { ref: [ '$B', 'author_ID' ] },
            '=',
            { ref: [ '$a', 'ID' ] },
            'and',
            { ref: [ '$B', 'stock' ] },
            'between',
            { val: 50 },
            'and',
            { val: 100 }
          ]
        }
      }
    ]
  }
}

In an expand construction (a correlated subquery example)

When the infix filter is used in the context of an expand (which will result in a nested result set) then the filter (stock > 10) is pushed down into the context of the subquery constructed for the expand, and added to the correlated subquery's conditions.

> cds.ql`
  SELECT from ${Authors}
  { books[stock > 10] { title } }
  `.forSQL()
cds.ql {
  SELECT: {
    from: { ref: [ 'sap.capire.bookshop.Authors' ], as: '$A' },
    columns: [
      {
        SELECT: {
          from: { ref: [ 'sap.capire.bookshop.Books' ], as: '$b' },
          columns: [ { ref: [ '$b', 'title' ] } ],
          expand: true,
          one: false,
          where: [
            { ref: [ '$A', 'ID' ] },
            '=',
            { ref: [ '$b', 'author_ID' ] },
            'and',
            { ref: [ '$b', 'stock' ] },
            '>',
            { val: 10 }
          ]
        },
        as: 'books'
      }
    ]
  }
}

Expressiveness in infix filters

Rounding off this episode at around 57:13 Patrice shows how expressive infix filters can be, with:

> cds.ql`
  SELECT from ${Books}[
    stock between 50 and 100
    and exists genre[name = 'Fantasy']
  ]:author
  { name }
  `.forSQL()
cds.ql {
  SELECT: {
    from: { ref: [ 'sap.capire.bookshop.Authors' ], as: '$a' },
    columns: [ { ref: [ '$a', 'name' ] } ],
    where: [
      'exists',
      {
        SELECT: {
          from: { ref: [ 'sap.capire.bookshop.Books' ], as: '$B' },
          columns: [ { val: 1 } ],
          where: [
            { ref: [ '$B', 'author_ID' ] },
            '=',
            { ref: [ '$a', 'ID' ] },
            'and',
            { ref: [ '$B', 'stock' ] },
            'between',
            { val: 50 },
            'and',
            { val: 100 },
            'and',
            'exists',
            {
              SELECT: {
                from: { ref: [ 'sap.capire.bookshop.Genres' ], as: '$g' },
                columns: [ { val: 1 } ],
                where: [
                  { ref: [ '$g', 'ID' ] },
                  '=',
                  { ref: [ '$B', 'genre_ID' ] },
                  'and',
                  { ref: [ '$g', 'name' ] },
                  '=',
                  { val: 'Fantasy' }
                ]
              }
            }
          ]
        }
      }
    ]
  }
}

We can see how the extended, chained set of conditions is materialised:

the stock condition becomes part of the EXISTS subquery's conditions
the genre condition becomes an additional correlated subquery which itself then becomes part of the EXISTS conditions too

That just about wraps it up for this episode, I hope you found some value in the video in these notes!

Further info

Genres, cuids and a bit of AWK

Footnotes

I find it interesting how we can use the > operator on dateOfBirth and supply only the first part of a value i.e. 1900, as if it were a string style comparison.
Later on in this episode at 54:48 I ask whether we can add such an order by in an infix filter at the schema level, and the answer helped me understand the difference between such "post-condition" filters and "restrictive". The answer was "no, not yet", but included an explanation which made a lot of sense - a restrictive filter (such as stock > 10) can be added to the WHERE condition in the subquery or JOIN, whereas a "post-condition" filter (such as order by name asc) cannot. The latter is only really possible right now at the query level.
Of course, this example assumes we do not want to, or cannot, add the annotation directly to the element where it occurs (@readonly e : String).
The language server based CDS formatter has inserted the spaces either side of the colon, which is a nice touch.

CDS expressions in CAP - notes on Part 5

2026-04-07T00:00:00Z

See the series post for an overview of all the episodes.

Introduction

00:00 Introduction and recap from last time.

06:55 Patrice jumps back in and visits the CXL documentation in Capire, which already by this point has even been updated and improved. He also briefly runs over some of the expressions and concepts we've covered thus far, including the CASE statement, predicates, association-like calculated elements (derived from existing associations) and more.

Combining CASE and concatenation operators

11:33 Revisiting my question last time about element references, Patrice expands the fullName example to combine some of the concepts with which we're now familiar:

the ternary operator (syntactic sugar for the CASE construct)
string concatenation (||)

fullName = academicTitle is not null
  ? academicTitle || ' ' || name
  : name;

This allows us to take Tolkien's "Professor" title into account 👍.

To show the result of this expression, Patrice runs a query¹ in the cds REPL:

> await SELECT
    .from(Authors)
    .columns('name', 'academicTitle', 'fullName')
[
  ...,
  {
    name: 'Brandon Sanderson',
    academicTitle: null,
    fullName: 'Brandon Sanderson'
  },
  {
    name: 'J. R. R. Tolkien',
    academicTitle: 'Prof.',
    fullName: 'Prof. J. R. R. Tolkien'
  }
]

If you want to copy this to try it out yourself, you can use the cds REPL's .editor feature to enter the multi-line construct (shown like this for better readability here).

Explicit and default types for calculated elements

14:01 I ask a question relating to the type declarations (or lack thereof) for the calculated elements. I came up with a very fanciful theory, only to be told that types are declared when the type is not String. In other words, if a type is not declared, then String is the default (for fullName here):

entity Authors : managed {
  key ID            : Integer;
      name          : String(111) @mandatory;
      address       : Association to Addresses;
      academicTitle : String(111);
      ...

      fullName = academicTitle is not null
        ? academicTitle || ' ' || name
        : name;

      isAlive       : Boolean = dateOfDeath is null ? true : false;
      age           : Integer = years_between(
        dateOfBirth, coalesce(dateOfDeath, current_date)
      );
}

Keep services simple

16:25 Then comes a great question from Neil, on hints, tips and best practices for complex models in the context of large data volumes.

In response, Patrice talks about how caution is needed when constructing definitions with large numbers of associations, especially when querying views that result from such definitions, where the queries are only to retrieve a small subset of data. To satisfy any query, a complex and possibly costly FROM clause needs to be processed by the underlying database, which when compared to the small query surface area, is then costly in comparison.

It's much better to keep service definitions simple and granular; think of services as reflectors of single domain problems, rather than representing the entire domain.

See also AXI004 Services are cheap.

This is also why it's always important to not only know about the power that we are able to wield, but also to know what happens behind the scenes. Two key reasons for this series!

Additionally, another member of the CAP team in the chat, Johannes Vogt, suggests employing DEBUG=sql as another way to see what's going on. At around 21:33 Patrice demonstrates this (with DEBUG=sql cds repl --run .).

Always consider EXISTS for checks across to-many relationships

24:10 Patrice concludes his wrap up by emphasising the importance of the EXISTS predicate. Not using this predicate results in likely unwanted duplicate records in the result set due to the LEFT JOIN that is used:

> await cds.ql`
  SELECT from Authors { fullName }
  where books.title like '%Mistborn%'
  `
[
  { fullName: 'Brandon Sanderson' },
  { fullName: 'Brandon Sanderson' }
]

Reformulating the above to use EXISTS plus an infix filter solves that, as a subquery (a "subselect") is used instead:

> await cds.ql`
  SELECT from Authors { fullName }
  where exists books[title like '%Mistborn%']
  `
[
  { fullName: 'Brandon Sanderson' }
]

See the section A look at the EXISTS predicate in the notes to the previous episode for more detail.

Simplicity from shifting left

27:00 Revisiting the association-like calculated element from last time, we get to meditate a bit more² on how shifting left brings about simplicity.

It would not be out of the ordinary to construct and execute a query like this:

> await cds.ql`
  SELECT from Authors { fullName }
  where exists books[stock > 170]
  `
[
  { fullName: 'Richard Carpenter' },
  { fullName: 'Prof. J. R. R. Tolkien' }
]

This is already an "accomplished" query, using the very constructions we looked at just earlier. However, if we shift this condition left, moving it from the query to our CDS model:

entity Authors : managed {
  key ID            : Integer;
      name          : String(111) @mandatory;
      ...

      books         : Association to many Books
                        on books.author = $self;

      nonSeller = books[ stock > 170 ];
      ...
}

then we define it once, can identify and test it once, and consumers have a convenient semantic shortcut to what the domain modelling process has defined.

Moreover, let's just take a second to boggle at the simplicity of the expression that is then available to us in query construction:

> await cds.ql`
  SELECT from Authors { fullName }
  where exists nonSeller
  `
[
  { fullName: 'Richard Carpenter' },
  { fullName: 'Prof. J. R. R. Tolkien' }
]

Digging into the condition

Here it is: where exists nonSeller. Now that is simple. How exactly are we checking for existence here? And what exactly are we checking anyway? An association? Well, yes, but this is where the term "association-like calculated element" fits much better.

At 28:31 Patrice takes a moment to look under the hood at this, so we can understand better what is going on.

First, the CQN shows us that the the target of the EXISTS is an expression { ref: [ 'nonSeller' ] }:

> q = cds.ql`
  SELECT from ${Authors} { fullName }
  where exists nonSeller
  `
cds.ql {
  SELECT: {
    from: { ref: [ 'sap.capire.bookshop.Authors' ] },
    columns: [ { ref: [ 'fullName' ] } ],
    where: [ 'exists', { ref: [ 'nonSeller' ] } ]
  }
}

Moving to the normalised "CAP-style SQL", with forSQL()³, we see somewhat of an explosion:

> q.forSQL()
cds.ql {
  SELECT: {
    from: { ref: [ 'sap.capire.bookshop.Authors' ], as: 'Authors' },
    columns: [
      {
        xpr: [
          'case',
          'when',
          { ref: [ 'Authors', 'academicTitle' ] },
          'is',
          'not',
          'null',
          'then',
          { ref: [ 'Authors', 'academicTitle' ] },
          '||',
          { val: ' ' },
          '||',
          { ref: [ 'Authors', 'name' ] },
          'else',
          { ref: [ 'Authors', 'name' ] },
          'end'
        ],
        as: 'fullName'
      }
    ],
    where: [
      'exists',
      {
        SELECT: {
          from: { ref: [ 'sap.capire.bookshop.Books' ], as: 'nonSeller' },
          columns: [ { val: 1 } ],
          where: [
            {
              xpr: [
                { ref: [ 'nonSeller', 'author_ID' ] },
                '=',
                { ref: [ 'Authors', 'ID' ] }
              ]
            },
            'and',
            { xpr: [ { ref: [ 'nonSeller', 'stock' ] }, '>', { val: 170 } ] }
          ]
        }
      }
    ]
  }
}

For the sake of this discussion, we can ignore the first part, specifically the value of columns, as that is for the fullName element. It's the WHERE clause that is of interest, and where the answers to the questions earlier start to appear.

You may wish to refer to the section A look at the EXISTS predicate from the notes to the previous episode for a detailed analysis, but revisiting this allows us to stare at the construct a little more, and see how it fits together - the subquery includes both conditions:

one from the nonSeller definition
one from the "donor" books definition

and either returns something (1)⁴ or nothing, which is why we can treat it almost like a Boolean.

31:30 Patrice makes a point about the SQL that is ultimately produced here, in that it's perfectly possible to construct that SQL yourself, manually. But who would want to do that, also taking into account the nuances of different database SQL dialects?

Moreover, one could consider taking one step back and writing the CAP-style SQL manually instead, using cds.ql facilities⁵. But for everyday development, this is extra effort that is not required, when compared to the power & expressiveness of CQL combined with CXL.

Using a path expression in the column list

At 32:36 Patrice continues the exploration of the nonSeller association-like calculated element, adding a couple of path expressions (nonSeller.stock and nonSeller.title) to the column list of the query:

> q = cds.ql`
  SELECT from ${Authors}
  { fullName, nonSeller.stock, nonSeller.title }
  where exists nonSeller
  `
cds.ql {
  SELECT: {
    from: { ref: [ 'sap.capire.bookshop.Authors' ] },
    columns: [
      { ref: [ 'fullName' ] },
      { ref: [ 'nonSeller', 'stock' ] },
      { ref: [ 'nonSeller', 'title' ] }
    ],
    where: [ 'exists', { ref: [ 'nonSeller' ] } ]
  }
}

The challenge here is that the introduction of the path expression into this query causes a flat list to be produced, with multiple entries for the authors involved:

> await q
[
  {
    fullName: 'Richard Carpenter',
    nonSeller_stock: 187,
    nonSeller_title: 'Catweazle'
  },
  {
    fullName: 'Prof. J. R. R. Tolkien',
    nonSeller_stock: 178,
    nonSeller_title: 'Beren and Lúthien'
  },
  {
    fullName: 'Prof. J. R. R. Tolkien',
    nonSeller_stock: 203,
    nonSeller_title: 'The Children of Húrin'
  },
  {
    fullName: 'Prof. J. R. R. Tolkien',
    nonSeller_stock: 195,
    nonSeller_title: 'The Fall of Gondolin'
  },
  {
    fullName: 'Prof. J. R. R. Tolkien',
    nonSeller_stock: 189,
    nonSeller_title: 'Unfinished Tales'
  }
]

Now we know about forSQL() and toSQL(), we can comfortably look behind the scenes, as Patrice does at 34:17, to understand why:

> q.forSQL()
cds.ql {
  SELECT: {
    from: {
      join: 'left',
      args: [
        { ref: [ 'sap.capire.bookshop.Authors' ], as: 'Authors' },
        { ref: [ 'sap.capire.bookshop.Books' ], as: 'nonSeller' }
      ],
      on: [
        {
          xpr: [
            { ref: [ 'nonSeller', 'author_ID' ] },
            '=',
            { ref: [ 'Authors', 'ID' ] }
          ]
        },
        'and',
        {
          xpr: [ { ref: [ 'nonSeller', 'stock' ] }, '>', { val: 170 } ]
        }
      ]
    },
    columns: [
      {
        xpr: [
          'case',
          'when',
          { ref: [ 'Authors', 'academicTitle' ] },
          'is',
          'not',
          'null',
          'then',
          { ref: [ 'Authors', 'academicTitle' ] },
          '||',
          { val: ' ' },
          '||',
          { ref: [ 'Authors', 'name' ] },
          'else',
          { ref: [ 'Authors', 'name' ] },
          'end'
        ],
        as: 'fullName'
      },
      { ref: [ 'nonSeller', 'stock' ], as: 'nonSeller_stock' },
      { ref: [ 'nonSeller', 'title' ], as: 'nonSeller_title' }
    ],
    where: [
      'exists',
      {
        SELECT: {
          from: { ref: [ 'sap.capire.bookshop.Books' ], as: 'nonSeller2' },
          columns: [ { val: 1 } ],
          where: [
            {
              xpr: [
                { ref: [ 'nonSeller2', 'author_ID' ] },
                '=',
                { ref: [ 'Authors', 'ID' ] }
              ]
            },
            'and',
            { xpr: [ { ref: [ 'nonSeller2', 'stock' ] }, '>', { val: 170 } ] }
          ]
        }
      }
    ],
    expand: 'root'
  }
}

Patrice points out the alias nonSeller2 - the term nonSeller has already been used for the alias for Books in the LEFT JOIN constructed due to the path expressions we added to the column list.

37:35 In answering a question I asked about this, Patrice explains the query plan here, which is to:

Filter the entire set of authors down to those satisfying the non-seller predicate condition
For that subset of authors, a LEFT JOIN is made to the books

The duplicate author names in the result set are because of this LEFT JOIN which brings about a flattening, essentially a consequence of the implicit requirement for a LEFT JOIN contradicting or invalidating the point of the subquery.

One could have just as well constructed the query like this, which Patrice shows at 38:33:

> q = cds.ql`
  SELECT from ${Authors}
  { fullName, nonSeller.stock, nonSeller.title }
  where nonSeller.title is not null
  `
cds.ql {
  SELECT: {
    from: { ref: [ 'sap.capire.bookshop.Authors' ] },
    columns: [
      { ref: [ 'fullName' ] },
      { ref: [ 'nonSeller', 'stock' ] },
      { ref: [ 'nonSeller', 'title' ] }
    ],
    where: [ { ref: [ 'nonSeller', 'title' ] }, 'is', 'not', 'null' ]
  }
}

The result set is the same as before.

Nested expands

39:50 However, there's more to life than flattened lists! Returning to the original query with the EXISTS predicate, we can avoid the flattening and repetition (which we now know is caused by the LEFT JOIN, due in turn to the path expression traversal requirements).

How? By using the power of nested expands:

> q = cds.ql`
  SELECT from ${Authors}
  { fullName, nonSeller { title, stock } }
  where exists nonSeller
  `
cds.ql {
  SELECT: {
    from: { ref: [ 'sap.capire.bookshop.Authors' ] },
    columns: [
      { ref: [ 'fullName' ] },
      {
        ref: [ 'nonSeller' ],
        expand: [ { ref: [ 'title' ] }, { ref: [ 'stock' ] } ]
      }
    ],
    where: [ 'exists', { ref: [ 'nonSeller' ] } ]
  }
}

This brings back a result set that is definitely not flattened:

> await q
[
  {
    fullName: 'Richard Carpenter',
    nonSeller: [ { title: 'Catweazle', stock: 187 } ]
  },
  {
    fullName: 'Prof. J. R. R. Tolkien',
    nonSeller: [
      { title: 'Unfinished Tales', stock: 189 },
      { title: 'The Children of Húrin', stock: 203 },
      { title: 'Beren and Lúthien', stock: 178 },
      { title: 'The Fall of Gondolin', stock: 195 }
    ]
  }
]

... it's structured!

This seems a little extraordinary, given the nature of SQL generally, and the native inability to store an entire set of data into a single column in particular. This extraordinariness is made possible due to the modern database adapters, re-written and introduced in CAP major version 8.

42:50 Patrice explains how this was achieved before those new database adapters: such queries were realised by a combination of SQL and also runtime logic, perhaps a bit like we approached similar query tasks in ABAP by using internal tables and custom logic controlled execution of various SELECT statements, back in the day.

Looking back at the Release Notes for Jun 2024, introducing CAP 8, we can see exactly what this is about in the New Database Services (GA) section, in particular:

Various optimizations like using database-native JSON functions for deep queries in single roundtrips, user-defined functions and more, to push data-processing tasks down to the database (→ improves utilization).

What do these "database-native JSON functions" look like? Well, we've seen them in passing before, but we can work our way towards them by following the now-familiar path, going from the CQN, to the CAP-style SQL, and ultimately to the native SQL.

First, the CAP-style SQL (with the expression for the fullName construction and the subquery for the exists nonSeller both elided for brevity):

> q.forSQL()
cds.ql {
  SELECT: {
    from: { ref: [ 'sap.capire.bookshop.Authors' ], as: 'Authors' },
    columns: [
      {
        xpr: [
          ...
        ],
        as: 'fullName'
      },
      {
        SELECT: {
          from: { ref: [ 'sap.capire.bookshop.Books' ], as: 'nonSeller2' },
          columns: [
            { ref: [ 'nonSeller2', 'stock' ] },
            { ref: [ 'nonSeller2', 'title' ] }
          ],
          expand: true,
          one: false,
          where: [
            {
              xpr: [
                { ref: [ 'Authors', 'ID' ] },
                '=',
                { ref: [ 'nonSeller2', 'author_ID' ] }
              ]
            },
            'and',
            { xpr: [ { ref: [ 'nonSeller2', 'stock' ] }, '>', { val: 170 } ] }
          ]
        },
        as: 'nonSeller'
      }
    ],
    where: [
      'exists',
      {
        ...
      }
    ]
  }
}

Note that there's no LEFT JOIN that we've seen employed previously (before we introduced the nested expand). Instead, alongside the fullName column, there's now a second column in the outermost (main) query, which is a subquery, specifically a SELECT on Books.

The WHERE clause in this subquery should look familiar, and serves to correlate the IDs of the Authors from the main query as well as restricting the result set according to the stock values.

48:16 In answer to my question at this point, Patrice tells us that these expands are similar to postfix projections and the concepts are shared.

Here follows an example, which shows:

the smart * selector
a couple of alias declarations (one for the actual association, and one for one of the structure elements)
omission of elements via an excluding clause

> await cds.ql`
  SELECT from ${Authors}
  {
    fullName,
    nonSeller as booksNotSellingWell
      { *, title as bookName }
      excluding { createdBy, modifiedBy, ID, descr }
  }
  where exists nonSeller
  `
[
  {
    fullName: 'Richard Carpenter',
    nonSeller: [
      {
        createdAt: '2026-04-04T10:29:20.859Z',
        modifiedAt: '2026-04-04T10:29:20.859Z',
        title: 'Catweazle',
        author_ID: 170,
        genre_ID: 13,
        stock: 187,
        price: 150,
        currency_code: 'JPY',
        bookName: 'Catweazle'
      }
    ]
  },
  {
    fullName: 'Prof. J. R. R. Tolkien',
    nonSeller: [
      {
        createdAt: '2026-04-04T10:29:20.859Z',
        modifiedAt: '2026-04-04T10:29:20.859Z',
        title: 'Unfinished Tales',
        author_ID: 201,
        genre_ID: 13,
        stock: 189,
        price: 13.99,
        currency_code: 'GBP',
        bookName: 'Unfinished Tales'
      },
      {
        createdAt: '2026-04-04T10:29:20.859Z',
        modifiedAt: '2026-04-04T10:29:20.859Z',
        title: 'The Children of Húrin',
        author_ID: 201,
        genre_ID: 13,
        stock: 203,
        price: 13.99,
        currency_code: 'GBP',
        bookName: 'The Children of Húrin'
      },
      ...
    ]
  }
]

At 50:12 I make an observation about nesting depth, to which Patrice responds by extending the example to add genre information, which I will do here to this example:

> await cds.ql`
  SELECT from ${Authors}
  {
    fullName,
    nonSeller as booksNotSellingWell
      { *, title as bookName, genre { * } }
      excluding { createdBy, modifiedBy, ID, descr }
  }
  where exists nonSeller
  `

The result set is suitably extended, here's what a typical book structure looks like now:

{
  createdAt: '2026-04-04T10:29:20.859Z',
  modifiedAt: '2026-04-04T10:29:20.859Z',
  title: 'The Fall of Gondolin',
  author_ID: 201,
  genre_ID: 13,
  stock: 195,
  price: 13.99,
  currency_code: 'GBP',
  bookName: 'The Fall of Gondolin',
  genre: {
    name: 'Fantasy',
    descr: null,
    ID: 13,
    parent_ID: 10
  }
}

Note also, as Patrice points out, that before CAP 8, this would have been realised by multiple (three, in fact) separate calls to the database layer, coordinated by logic in the runtime, and then the results stitched together before being returned as a contiguous set. These three calls can be seen as three nested SELECT statements in the query's intermediate format⁶.

With the new database adapters, there's only a single call to the database layer, and no coordination or combination logic required at runtime.

JSON functions in SQL

53:18 Based on this query that Patrice was working with:

> q = cds.ql`
  SELECT from ${Authors}
  {
    fullName,
    nonSeller
    {
      title as book,
      stock,
      genre
      {
        *
      }
    }
  }
  where exists nonSeller
  `

here's what the actual database native (SQLite in this particular example) SQL looks like⁷:

SELECT
  case
    when Authors.academicTitle is not null then Authors.academicTitle || ? || Authors.name
    else Authors.name
  end as fullName,
  (
    SELECT
      jsonb_group_array (
        jsonb_insert (
          '{}',
          '$."book"',
          book,
          '$."stock"',
          stock,
          '$."genre"',
          genre - > '$'
        )
      ) as _json_
    FROM
      (
        SELECT
          nonSeller2.title as book,
          nonSeller2.stock,
          (
            SELECT
              json_insert (
                '{}',
                '$."name"',
                name,
                '$."descr"',
                descr,
                '$."ID"',
                ID,
                '$."parent_ID"',
                parent_ID
              ) as _json_
            FROM
              (
                SELECT
                  genre.name,
                  genre.descr,
                  genre.ID,
                  genre.parent_ID
                FROM
                  sap_capire_bookshop_Genres as genre
                WHERE
                  "nonSeller2".genre_ID = genre.ID
                LIMIT
                  ?
              )
          ) as genre
        FROM
          sap_capire_bookshop_Books as "nonSeller2"
        WHERE
          (Authors.ID = "nonSeller2".author_ID)
          and ("nonSeller2".stock > ?)
      )
  ) as nonSeller
FROM
  sap_capire_bookshop_Authors as Authors
WHERE
  exists (
    SELECT
      1 as "1"
    FROM
      sap_capire_bookshop_Books as nonSeller
    WHERE
      (nonSeller.author_ID = Authors.ID)
      and (nonSeller.stock > ?)
  )

Here are some initial notes on this (single!) SQL statement:

the outermost SELECT is on the Authors
it is constrained by the WHERE clause that represents the non-seller subquery with which we are familiar
the construction of the fullName in SQL here is extremely similar to the CXL CASE expression

Then come the nested subqueries. But wait, that's a lot more SELECTs that we expected! That's because of the interleaving of JSON functions, from which comes the power and ability to push down such complex queries directly and solely to the database layer.

Essentially, we see the use here of:

the aggregate function jsonb_group_array
the function json_insert to construct JSON objects (to be aggregated)

The juxtaposition (pairing, almost) of the JSON functions and the corresponding subquery SELECT statements is not accidental. They have been generated exactly like this to be able to build the deeply nested structure required, otherwise impossible in SQL without such JSON facilities ... combined with the ability to stringify complex JSON structures into scalar values (large strings!).

Further info

Patrice also has some great notes for this part.

Footnotes

The query is constructed using the fluent API style, rather than what we've mostly employed, which has been by writing queries in tagged template literals. See the Constructing Queries section of the Querying in JavaScript topic in Capire for more information. Additionally, I used the columns() method to restrict the data set returned.
We considered this briefly in part 3 of this series - see Adding a calculated element in the notes.
It's for the benefit of this method that we're using ${Authors} rather than Authors in the query template string, so that the transformation function will work properly.

Note that this value of 1 is not the SQLite Boolean value for true. This is the normalised neutral SQL (from forSQL()) rather than the database specific SQL. If we were to ask for the database specific SQL when connected to HANA (instead of SQLite here):

echo 'cds.ql`select from ${Authors} { fullName } where exists nonSeller`.toSQL().sql'
| cds bind --exec --profile hybrid -- cds repl --run .

we'd see this 1 value too:

SELECT
  case
    when Authors.academicTitle is not null then Authors.academicTitle || ? || Authors.name
    else Authors.name
  end as "fullName"
FROM
  sap_capire_bookshop_Authors as Authors
WHERE
  exists (
    SELECT
      1 as "1"
    FROM
      sap_capire_bookshop_Books as nonSeller
    WHERE
      (nonSeller.author_ID = Authors.ID)
      and (nonSeller.stock > ?)
  )

cds.ql contains a whole host of facilities for this:

> Object.keys(cds.ql)
[
  'Query',   'SELECT', 'INSERT',
  'UPSERT',  'UPDATE', 'DELETE',
  'CREATE',  'DROP',   'predicate',
  'columns', 'ref',    'val',
  'xpr',     'expr',   'list',
  'func',    'nested', 'expand',
  'inline',  'where',  'orderBy',
  'orders',  'limit',  'clone'
]

Here's the intermediate format for this query, showing the SELECT statements nested as columns:

cds.ql {
  SELECT: {
    from: { ref: [ 'sap.capire.bookshop.Authors' ], as: 'Authors' },
    columns: [
      {
        xpr: [
          ...
        ],
        as: 'fullName'
      },
      {
        SELECT: {
          excluding: [ 'createdBy', 'modifiedBy', 'ID', 'descr' ],
          from: { ref: [ 'sap.capire.bookshop.Books' ], as: 'nonSeller2' },
          columns: [
            { ref: [ 'nonSeller2', 'createdAt' ] },
            { ref: [ 'nonSeller2', 'modifiedAt' ] },
            { ref: [ 'nonSeller2', 'title' ] },
            { ref: [ 'nonSeller2', 'author_ID' ] },
            { ref: [ 'nonSeller2', 'genre_ID' ] },
            { ref: [ 'nonSeller2', 'stock' ] },
            { ref: [ 'nonSeller2', 'price' ] },
            { ref: [ 'nonSeller2', 'currency_code' ] },
            { ref: [ 'nonSeller2', 'title' ], as: 'bookName' },
            {
              SELECT: {
                from: { ref: [ 'sap.capire.bookshop.Genres' ], as: 'genre' },
                columns: [
                  { ref: [ 'genre', 'name' ] },
                  { ref: [ 'genre', 'descr' ] },
                  { ref: [ 'genre', 'ID' ] },
                  { ref: [ 'genre', 'parent_ID' ] }
                ],
                expand: true,
                one: true,
                where: [
                  { ref: [ 'nonSeller2', 'genre_ID' ] },
                  '=',
                  { ref: [ 'genre', 'ID' ] }
                ]
              },
              as: 'genre'
            }
          ],
          expand: true,
          one: false,
          where: [
            {
              xpr: [
                { ref: [ 'Authors', 'ID' ] },
                '=',
                { ref: [ 'nonSeller2', 'author_ID' ] }
              ]
            },
            'and',
            { xpr: [ { ref: [ 'nonSeller2', 'stock' ] }, '>', { val: 170 } ] }
          ]
        },
        as: 'nonSeller'
      }
    ],
    where: [
      'exists',
      {
        SELECT: {
          from: { ref: [ 'sap.capire.bookshop.Books' ], as: 'nonSeller' },
          columns: [ { val: 1 } ],
          where: [
            {
              xpr: [
                { ref: [ 'nonSeller', 'author_ID' ] },
                '=',
                { ref: [ 'Authors', 'ID' ] }
              ]
            },
            'and',
            { xpr: [ { ref: [ 'nonSeller', 'stock' ] }, '>', { val: 170 } ] }
          ]
        }
      }
    ]
  }
}

The outermost JSON function has been omitted here, as it is less important to stare at right now.

CDS expressions in CAP - notes on Part 4

2026-03-27T00:00:00Z

See the series post for an overview of all the episodes.

Introduction

00:00 Introduction and recap from last time.

07:00 Patrice revisits the syntax diagram and starts to explain the insignificant-looking but very significant (in terms of power and utility) "ref" box in that diagram, which is the start of our journey to understand path expressions.

In fact, the box label "ref" at the time this episode was being broadcast has now been replaced with "path expression" in the latest incarnation of the syntax diagram in Capire:

Selecting this box leads to the Path Expressions (ref) section of the CXL topic.

Some syntactic sugar for the case-when-then-else-end expression

10:08 Patrice revisits the CASE ... WHEN ... THEN ... ELSE ... END expression, which we looked at first in part 2 but for which there's some nice syntactic sugar.

He starts with passing a simple CXL example to the context-free parser (cds.parse.expr), which emits the corresponding CXN:

> cds.parse.expr`
  case when stock > 10 then 'non-seller' else 'sells quickly' end
  `
{
  xpr: [
    'case',
    'when',
    { ref: [ 'stock' ] },
    '>',
    { val: 10 },
    'then',
    { val: 'non-seller' },
    'else',
    { val: 'sells quickly' },
    'end'
  ]
}

Using the syntactic sugar, Patrice then shows us that we can write the same expression using a construct familiar in JavaScript and other languages:

> cds.parse.expr`
  stock > 10 ? 'non seller' : 'sells quickly'
  `
{
  xpr: [
    'case',
    'when',
    { ref: [ 'stock' ] },
    '>',
    { val: 10 },
    'then',
    { val: 'non seller' },
    'else',
    { val: 'sells quickly' },
    'end'
  ]
}

This version of the CXL expression is actually exactly the same in the internal (CXN) machine-readable form. Not only that, but this ? : is nestable too:

> cds.parse.expr`
  stock > 10
  ? (price > 50 ? 'expensive non seller' : 'cheap non seller')
  : 'sells quickly'
  `
{
  xpr: [
    'case',
    'when',
    { ref: [ 'stock' ] },
    '>',
    { val: 10 },
    'then',
    {
      xpr: [
        'case',
        'when',
        { ref: [ 'price' ] },
        '>',
        { val: 50 },
        'then',
        { val: 'expensive non seller' },
        'else',
        { val: 'cheap non seller' },
        'end'
      ]
    },
    'else',
    { val: 'sells quickly' },
    'end'
  ]
}

A closer look at the predicate expressions

15:15 The last thing that Patrice talks about before jumping into path expressions is the collection of predicates, which we very briefly noted in the previous episode.

First, we look at the BETWEEN predicate, by first considering an compound expression that describes a closed interval, to check for stock between 10 and 30:

> cds.parse.expr`
  stock >= 10 and stock <= 30
  `
{
  xpr: [
    { ref: [ 'stock' ] },
    '>=',
    { val: 10 },
    'and',
    { ref: [ 'stock' ] },
    '<=',
    { val: 30 }
  ]
}

Here we have two binary operator based expressions (with the comparison operators >= and <=) that are joined with (and become the operands for) another binary operator, the logical operator and.

In contrast, there's the range checking operator between, which we can use instead:

> cds.parse.expr`
  stock between 12 and 34
  `
{
  xpr: [ { ref: [ 'stock' ] }, 'between', { val: 12 }, 'and', { val: 34 } ]
}

Not only is this a single expression, it is also much neater and (arguably) easier to read as well as write.

Here's that expression in action, in a query:

> await cds.ql`
  select from Books { title, stock } where stock between 12 and 34
  `
[
  { title: 'Mistborn: The Final Empire', stock: 12 },
  { title: 'The Two Towers', stock: 14 }
]

There's also the not variant which is also available.

20:30 Next up is IN, which Patrice first demonstrates in an abstract way with:

> cds.parse.expr` (1, 2, 3) in (select ID from Books)`
{
  xpr: [
    { list: [ { val: 1 }, { val: 2 }, { val: 3 } ] },
    'in',
    {
      SELECT: {
        from: { ref: [ 'Books' ] },
        columns: [ { ref: [ 'ID' ] } ]
      }
    }
  ]
}

but then goes on to show the operator employed in a more practical example:

> q = cds.ql`
  select from ${Books} { title, stock }
  where (author.ID) in
    (select ID from ${Authors} where dateOfDeath is null)
  `
cds.ql {
  SELECT: {
    from: { ref: [ 'sap.capire.bookshop.Books' ] },
    columns: [ { ref: [ 'title' ] }, { ref: [ 'stock' ] } ],
    where: [
      { ref: [ 'author', 'ID' ] },
      'in',
      {
        SELECT: {
          from: { ref: [ 'sap.capire.bookshop.Authors' ] },
          columns: [ { ref: [ 'ID' ] } ],
          where: [ { ref: [ 'dateOfDeath' ] }, 'is', 'null' ]
        }
      }
    ]
  }
}

Here, the ID of the author association (via author_ID, effectively) is matched to the set of values from the (select ID from ...) subquery:

> await q
[
  { title: 'Mistborn: The Final Empire', stock: 12 },
  { title: 'The Well of Ascension', stock: 8 },
  { title: 'The Hero of Ages', stock: 5 },
  { title: 'The Alloy of Law', stock: 67 },
  { title: 'Shadows of Self', stock: 89 },
  { title: 'The Bands of Mourning', stock: 134 },
  { title: 'The Lost Metal', stock: 156 },
  { title: 'Mistborn: Secret History', stock: 98 },
  { title: 'The Way of Kings', stock: 7 },
  { title: 'Words of Radiance', stock: 4 },
  { title: 'Edgedancer', stock: 125 },
  { title: 'Oathbringer', stock: 3 },
  { title: 'Dawnshard', stock: 142 },
  { title: 'Rhythm of War', stock: 6 },
  { title: 'Wind and Truth', stock: 2 }
]

Moreover, the expression (author.ID) could have been written as author.ID i.e. as a path expression, rather than a path expression within a list context, which Patrice explains at 24:45, and also clarifies at 27:59 that the subquery is indeed an expression (it is, according to the syntax diagram) - a "query expression".

A first look at path expressions

There's also the EXISTS predicate which we will want to take a look at, but because this is most often found in use with path expressions, Patrice takes us on our first excursion to explore their power and utility, starting at 29:30, with a simple example (some results omitted for brevity here):

> q = cds.ql`SELECT from ${Authors} {name as author, books.title }`
cds.ql {
  SELECT: {
    from: { ref: [ 'sap.capire.bookshop.Authors' ] },
    columns: [
      { ref: [ 'name' ], as: 'author' },
      { ref: [ 'books', 'title' ] }
    ]
  }
}
> await q
[
  { author: 'Emily Brontë', books_title: 'Wuthering Heights' },
  { author: 'Charlotte Brontë', books_title: 'Jane Eyre' },
  { author: 'Edgar Allen Poe', books_title: 'Eleonora' },
  { author: 'Edgar Allen Poe', books_title: 'The Raven' },
  { author: 'Richard Carpenter', books_title: 'Catweazle' },
  { author: 'Brandon Sanderson', books_title: 'Dawnshard' },
  { author: 'Brandon Sanderson', books_title: '...' },
  { author: 'Brandon Sanderson', books_title: 'Words of Radiance' },
  { author: 'J. R. R. Tolkien', books_title: 'Beren and Lúthien' },
  { author: 'J. R. R. Tolkien', books_title: '...' },
  { author: 'J. R. R. Tolkien', books_title: 'Unfinished Tales' }
]

This is a "flat" list, i.e. all of the authors + book combinations.

To help our understanding, Patrice then highlights the diagram specifically for path expressions, which looks like this:

A brief digression on the term "forward-declared join"

At this point (32:30) I'm unable to resist surfacing a phrase that is also used in this context, and that is "forward-declared join", which Daniel Hutzel and I touched upon in part 9 of The Art and Science of CAP. Patrice nicely explains what this term is, and how the concept it represents, is present in the model that we're using, for example in the definition of the Authors entity (some elements omitted here for brevity):

entity Authors : managed {
  key ID            : Integer;
      name          : String(111) @mandatory;
      books         : Association to many Books
                        on books.author = $self;
}

The "on condition" for the books element definition is a sign that some correlation is going to be involved (as Patrice rightly points out, correlation is at the heart of any RDBMS); what's more, that correlation connecting Authors to Books is defined ahead of time, before any actual manifestation of the JOIN mechanism that will be required to realise the correlation.

Digging deeper into path expressions

Turning back to the path expression we had in SELECT from ${Authors} {name as author, books.title }, Patrice had pointed out that this query is not directly possible (in its current, simple form) in SQL, as books.title traverses a path from one entity to another.

Indeed, at 36:07 we see what this query is going to become by using the forSQL() method on the query object, where we see a LEFT JOIN is planned in the SELECT specification, in this SQL-style CQN version of the query:

> q.forSQL()
cds.ql {
  SELECT: {
    from: {
      join: 'left',
      args: [
        { ref: [ 'sap.capire.bookshop.Authors' ], as: 'Authors' },
        { ref: [ 'sap.capire.bookshop.Books' ], as: 'books' }
      ],
      on: [
        { ref: [ 'books', 'author_ID' ] },
        '=',
        { ref: [ 'Authors' , 'ID' ] }
      ]
    },
    columns: [
      { ref: [ 'Authors' , 'name' ], as: 'author' },
      { ref: [ 'books', 'title' ], as: 'books_title' }
    ]
  }
}

Note that the name of the association element books becomes the alias for the target of the traversal.

So here we're seeing the time element of a forward-declared join i.e. a path expression, in that the join is manifested "just in time". By the way, with toSQL() we can see that the actual database engine specific SQL (SQLite in this case) looks like this (with the json_insert removed):

SELECT
  Authors.name as author,
  books.title as books_title
FROM
  sap_capire_bookshop_Authors as Authors
  left JOIN sap_capire_bookshop_Books as books
    ON books.author_ID = Authors.ID

Association-like calculated element

40:16 Building on this knowledge and what we learned last week, Patrice now adds a further element to the Authors entity definition, called nonSeller. And what a remarkable one! Here it is (again, with other elements removed for brevity):

entity Authors : managed {
  key ID            : Integer;
      name          : String(111) @mandatory;
      books         : Association to many Books
                        on books.author = $self;

      nonSeller = books[ stock > 100 ];

      isAlive       : Boolean = dateOfDeath is null ? true : false;
      age           : Integer = years_between(
        dateOfBirth, coalesce(
          dateOfDeath, current_date
        )
      );
}

This nonSeller element builds upon an existing association (books), and adds a condition in an infix filter ([stock > 100]). The syntax used for this definition basically follows the calculated element pattern. Patrice and I discussed the type, which is derived from the referenced element (books).

Additionally, the "on" condition in that referenced element is, for nonSeller, enhanced by the infix filter. This is exactly what's shown in the bottom part of the path expression syntax diagram shown earlier.

Essentially this new element nonSeller is still an association, and can therefore be used in path expressions just like books, as Patrice demonstrates at 43:46:

> q = cds.ql`SELECT from ${Authors} { name, nonSeller.title } `
cds.ql {
  SELECT: {
    from: { ref: [ 'sap.capire.bookshop.Authors' ] },
    columns: [ { ref: [ 'name' ] }, { ref: [ 'nonSeller', 'title' ] } ]
  }
}

Transforming the query into the normalised (intermediate) format, we see:

> q.forSQL()
cds.ql {
  SELECT: {
    from: {
      join: 'left',
      args: [
        { ref: [ 'sap.capire.bookshop.Authors' ], as: 'Authors' },
        { ref: [ 'sap.capire.bookshop.Books' ], as: 'nonSeller' }
      ],
      on: [
        {
          xpr: [
            { ref: [ 'nonSeller', 'author_ID' ] },
            '=',
            { ref: [ 'Authors', 'ID' ] }
          ]
        },
        'and',
        {
          xpr: [ { ref: [ 'nonSeller', 'stock' ] }, '>', { val: 100 } ]
        }
      ]
    },
    columns: [
      { ref: [ 'Authors', 'name' ] },
      { ref: [ 'nonSeller', 'title' ], as: 'nonSeller_title' }
    ]
  }
}

We can see that there's a left join planned, and there is a set of "on" conditions in an array, two expressions:

a correlation of the author ID across the two entities
a comparison operator based condition on the stock value

In both cases the entities are referred to by their aliases.

48:00 The dataset returned from this query looks like this:

  { name: 'Emily Brontë', nonSeller_title: null },
  { name: 'Charlotte Brontë', nonSeller_title: null },
  { name: 'Edgar Allen Poe', nonSeller_title: 'Eleonora' },
  { name: 'Edgar Allen Poe', nonSeller_title: 'The Raven' },
  { name: 'Richard Carpenter', nonSeller_title: 'Catweazle' },
  { name: 'Brandon Sanderson', nonSeller_title: 'Dawnshard' },
  { name: 'Brandon Sanderson', nonSeller_title: 'Edgedancer' },
  { name: 'Brandon Sanderson', nonSeller_title: 'The Bands of Mourning' },
  { name: 'Brandon Sanderson', nonSeller_title: 'The Lost Metal' },
  { name: 'J. R. R. Tolkien', nonSeller_title: 'Beren and Lúthien' },
  { name: 'J. R. R. Tolkien', nonSeller_title: 'The Children of Húrin' },
  { name: 'J. R. R. Tolkien', nonSeller_title: 'The Fall of Gondolin' },
  { name: 'J. R. R. Tolkien', nonSeller_title: 'The Silmarillion' },
  { name: 'J. R. R. Tolkien', nonSeller_title: 'Unfinished Tales' }
]

We're driving this query from the Authors, with a(n implicit) LEFT JOIN, which means that we are going to get all of the authors, including those (two of the Brontë sisters!) for which there are no "non sellers", i.e. no books with more than 100 in stock. We can deal with this in various ways, see the next section.

Element reference expressions

51:02 Based on a question from me going back to the CDL definition of nonSeller, and how it is basically a reference to an existing element (books), Patrice shows that this is a pattern that works generally. For example, we add an origin element that just points to placeOfBirth:

entity Authors : managed {
  key ID            : Integer;
      name          : String(111) @mandatory;
      address       : Association to Addresses;
      academicTitle : String(111);
      dateOfBirth   : Date;
      dateOfDeath   : Date;
      placeOfBirth  : String;
      placeOfDeath  : String;
      books         : Association to many Books
                        on books.author = $self;

      nonSeller = books[stock > 100];

      origin = placeOfBirth;

      isAlive       : Boolean = dateOfDeath is null ? true : false;
      age           : Integer = years_between(
        dateOfBirth, coalesce(
          dateOfDeath, current_date
        )
      );
}

Essentially this is just another calculated element, albeit a very simple one! Patrice gives another example:

name          : String(111) @mandatory;
academicTitle : String(10);
fullName = academicTitle || ' ' || name;

The type of fullName here is implicit but we can make it explicit:

fullName : String = academicTitle || ' ' || name;

like we have done for the isAlive and age elements.

A look at the EXISTS predicate

54:54 At this point we go back to the dataset that was produced from the query SELECT from ${Authors} { name, nonSeller.title }, in particular with the two author records with null for nonSeller_title. What if we wanted to exclude such authors, i.e. only include authors with non-sellers?

Patrice first shows us an option that is available to us, but one which we probably want to avoid, as it is a little clumsy, involves a somewhat technical approach:

> await cds.ql`select from ${Authors} { name } where nonSeller.ID is not null`
[
  { name: 'Edgar Allen Poe' },
  { name: 'Edgar Allen Poe' },
  { name: 'Richard Carpenter' },
  { name: 'Brandon Sanderson' },
  { name: 'Brandon Sanderson' },
  { name: 'Brandon Sanderson' },
  { name: 'Brandon Sanderson' },
  { name: 'J. R. R. Tolkien' },
  { name: 'J. R. R. Tolkien' },
  { name: 'J. R. R. Tolkien' },
  { name: 'J. R. R. Tolkien' },
  { name: 'J. R. R. Tolkien' }
]

Moreover, there is still an issue with the authors that do have non-sellers - there are duplicate records returned. This is a natural consequence of the to-many relationship:

entity Authors : managed {
  key ID            : Integer;
      name          : String(111) @mandatory;

      ...

      books         : Association to many Books
                        on books.author = $self;

      nonSeller = books[stock > 100];

      ...
}

and is essentially the result of a LEFT JOIN that traverses it.

A much better approach is to use the prefix operator EXISTS:

> await cds.ql`select from ${Authors} { name } where exists nonSeller`
[
  { name: 'Edgar Allen Poe' },
  { name: 'Richard Carpenter' },
  { name: 'Brandon Sanderson' },
  { name: 'J. R. R. Tolkien' }
]

This is almost magic! The expression used as the operand here is simply nonSeller, i.e. a reference to the association, to the relationship. It's the closest we can get to how we'd say it in English.

By now, we know how to look under the hood for this - we can trace the steps from this CQL and CXL ... through the various stages. First, here's what the "CAP style CQN" looks like:

> q = cds.ql`select from ${Authors} { name } where exists nonSeller`
cds.ql {
  SELECT: {
    from: { ref: [ 'sap.capire.bookshop.Authors' ] },
    columns: [ { ref: [ 'name' ] } ],
    where: [ 'exists', { ref: [ 'nonSeller' ] } ]
  }
}

Notice that the simple association name nonSeller here (as the target of exists, i.e. { ref: [ 'nonSeller' ]}) is perfectly valid according to the syntax diagram we looked at earlier in our first look at path expressions.

Next, we can see the normalised "CAP style SQL":

> q.forSQL()
cds.ql {
  SELECT: {
    from: { ref: [ 'sap.capire.bookshop.Authors' ], as: 'Authors' },
    columns: [ { ref: [ 'Authors', 'name' ] } ],
    where: [
      'exists',
      {
        SELECT: {
          from: { ref: [ 'sap.capire.bookshop.Books' ], as: 'nonSeller' },
          columns: [ { val: 1 } ],
          where: [
            {
              xpr: [
                { ref: [ 'nonSeller', 'author_ID' ] },
                '=',
                { ref: [ 'Authors', 'ID' ] }
              ]
            },
            'and',
            { xpr: [ { ref: [ 'nonSeller', 'stock' ] }, '>', { val: 100 } ] }
          ]
        }
      }
    ]
  }
}

Here are some observations on this:

the SELECT is on the Authors entity, as we expect
the WHERE clause is essentially the EXISTS predicate
the predicate's target (or focus) is a complete subquery
this subquery represents the association and is on the Books
a Boolean value is expected from this subquery
that value is represented by the dummy literal value { val: 1 }

What's not changed subquery's WHERE clause is exactly the same as what was in the on clause in the earlier query in the previous association-like calculated element section, representing the combined conditions of books.author = $self and stock > 100.

So moving from a LEFT JOIN to an EXISTS with a subquery ... moves us from duplicate data from the left part of the relationship, to unique values, as what's returned from this subquery is either something ({ val: 1 }) or nothing.

01:01:16 To underline the power and utility of this predicate, Patrice rounds the episode off with another example, contrasting the two approaches and result sets, based on the books from Brandon Sanderson, two of which include the word "Mistborn":

> await cds.ql`
  select title from ${Books} where author.name like '%Sanderson'
  `
[
  { title: 'Mistborn: The Final Empire' },
  { title: 'The Well of Ascension' },
  { title: 'The Lost Metal' },
  { title: 'Mistborn: Secret History' },
  { title: 'The Way of Kings' },
  { title: '...' },
  { title: 'Wind and Truth' }
]

First, the less convenient approach:

> q = cds.ql`
  select from ${Authors} { name }
  where books.title like '%Mistborn%'
  `
cds.ql {
  SELECT: {
    from: { ref: [ 'sap.capire.bookshop.Authors' ] },
    columns: [ { ref: [ 'name' ] } ],
    where: [ { ref: [ 'books', 'title' ] }, 'like', { val: '%Mistborn%' } ]
  }
}
> await q
[
  { name: 'Brandon Sanderson' },
  { name: 'Brandon Sanderson' }
]

And then the approach using EXISTS:

> q = cds.ql`
  select from ${Authors} { name }
  where exists books[title like '%Mistborn%']
  `
cds.ql {
  SELECT: {
    from: { ref: [ 'sap.capire.bookshop.Authors' ] },
    columns: [ { ref: [ 'name' ] } ],
    where: [
      'exists',
      {
        ref: [
          {
            id: 'books',
            where: [ { ref: [ 'title' ] }, 'like', { val: '%Mistborn%' } ]
          }
        ]
      }
    ]
  }
}
> await q
[
  { name: 'Brandon Sanderson' }
]

The EXISTS predicate is an essential item in your expression toolkit for checking across to-many relationships.

Further info

Patrice also has some great notes for this part.

CDS expressions in CAP - notes on Part 3

2026-03-23T00:00:00Z

See the series post for an overview of all the episodes.

Introduction

00:00 Introduction.

03:12 Patrice confirms that the constraints that we looked at in the previous part are checked in an "after" phase handler.

Revisiting core CXL building blocks

05:55 Patrice jumps into the cds REPL to revisit some of the CXL building blocks, such as literals:

> cds.parse.expr` 1 `
{ val: 1 }
> cds.parse.expr` true `
{ val: true }

unary operators:

> cds.parse.expr` +1 `
{ xpr: [ '+', { val: 1 } ] }

binary operators, including some that are common in programming languages, such as != which is translated to IS NOT in SQL:

> cds.ql`SELECT title from ${Books} where stock != null`.toSQL()
{
  sql: 'SELECT title AS "title" FROM (
    SELECT "$B".title
    FROM sap_capire_bookshop_Books as "$B"
    WHERE "$B".stock is not NULL
  )',
  values: []
}

This is simplified SQL without JSON, in a cds REPL session invoked thus:
CDS_FEATURES_SQL__SIMPLE__QUERIES=2 cds r -r .

Functions, CAST and predicates

09:58 Next up is a look at the function syntax, which Patrice shows with an example in the cds REPL, emphasising that the arguments are just expressions:

> cds.parse.expr` someFunction( (1+1), true ) `
{
  func: 'someFunction',
  args: [ { xpr: [ { val: 1 }, '+', { val: 1 } ] }, { val: true } ]
}

17:25 We take a brief look at CAST, a special function:

> cds.parse.expr` cast( 1 as String ) `
{ val: 1, cast: { type: 'String' } }

19:06 And then we look at the family of predicates available, such as [NOT] LIKE, IS [NOT] NULL, [NOT] BETWEEN ... AND ..., [NOT] IN ( ... ) and [NOT] EXISTS ....

Adding a calculated element

21:50 Patrice shows an example at the "db" level in the db/schema.cds file, by adding a calculated element (the sort that we might alternatively find in queries, i.e. in CQL) to the CDL definition of the Authors entity which currently looks like this:

entity Authors : managed {
  key ID            : Integer;
      name          : String(111) @mandatory;
      address       : Association to Addresses;
      academicTitle : String(111);
      dateOfBirth   : Date;
      dateOfDeath   : Date;
      placeOfBirth  : String;
      placeOfDeath  : String;
      books         : Association to many Books
                        on books.author = $self;
}

Adding a Boolean element isAlive can be done via an expression, in various ways, such as:

isAlive : Boolean = case
  when dateOfDeath is null then true
  else false
end;

Or with the concise ternary expression beloved of JavaScript (and other) programmers:

isAlive : Boolean = dateOfDeath is null ? true : false

Of course, this was purely to illustrate the ? : syntax; an even less verbose version would be:
isAlive : Boolean = dateOfDeath is null;

24:00 This calculated element is virtual, i.e. not manifested in the underlying table definition, which Patrice shows with:

cds compile -2 sql db/schema.cds

which emits this DDL, with no sign of the isAlive field:

CREATE TABLE sap_capire_bookshop_Authors (
  createdAt TIMESTAMP_TEXT,
  createdBy NVARCHAR(255),
  modifiedAt TIMESTAMP_TEXT,
  modifiedBy NVARCHAR(255),
  ID INTEGER NOT NULL,
  name NVARCHAR(111),
  address_ID INTEGER,
  academicTitle NVARCHAR(111),
  dateOfBirth DATE_TEXT,
  dateOfDeath DATE_TEXT,
  placeOfBirth NVARCHAR(255),
  placeOfDeath NVARCHAR(255),
  PRIMARY KEY(ID)
);

The calculated element only plays a role in queries, and in the context of views or projections. And we have one of those, in the form of the AdminService which looks like this:

using {sap.capire.bookshop as my} from '../db/schema';

service AdminService {
  entity Books   as projection on my.Books;
  entity Authors as projection on my.Authors;
}

// ...

Compiling srv/admin-service.cds:

cds compile -2 sql srv/admin-service.cds

shows us where this manifests, in the DDL for this Authors projection:

CREATE VIEW AdminService_Authors AS SELECT
  Authors_0.createdAt,
  Authors_0.createdBy,
  Authors_0.modifiedAt,
  Authors_0.modifiedBy,
  Authors_0.ID,
  Authors_0.name,
  Authors_0.address_ID,
  Authors_0.academicTitle,
  Authors_0.dateOfBirth,
  Authors_0.dateOfDeath,
  Authors_0.placeOfBirth,
  Authors_0.placeOfDeath,
  CASE WHEN Authors_0.dateOfDeath IS NULL THEN TRUE ELSE FALSE END AS isAlive
FROM sap_capire_bookshop_Authors AS Authors_0;

This, incidentally, is a great example of both shifting left and down, where the convenience mechanism that results in an isAlive Boolean value is not calculated by any requester (based on the dateOfDeath value that is available, once the dataset has been retrieved), or added to any transient dynamic queries, or even added as adornments to one or more service definitions that have projections on the authors data.

Instead, it is defined once, quietly and gently, at the "db" level, and reified where appropriate, and available automatically. As Patrice puts it a bit later on, this is "centralising our common expressions into one place - our domain model".

See the Further info section for links to more reading on this.

26:40 In the context of a question asked by VishalK, Patrice notes that there are two forms of calculated elements, on-read (as here) and also on-write.

28:20 Patrice illustrates the expression that has been constructed (in CXN) for this isAlive calculated element by looking directly at it in the cds REPL:

> Authors.elements['isAlive']
Boolean {
  '@Core.Computed': true,
  type: 'cds.Boolean',
  value: {
    xpr: [
      'case',
      'when',
      { ref: [ 'dateOfDeath' ] },
      'is',
      'null',
      'then',
      { val: true },
      'else',
      { val: false },
      'end'
    ]
  }
}

Using the calculated element in queries

29:26 Having looked at how the calculated element at the "db" level is manifested in the DDL for the AdminService's projection, Patrice now shows how it comes into play in a query, in the cds REPL:

> await cds.ql`
  SELECT from ${Books} { title, author.name }
  where author.isAlive = true
  `
[
  { title: 'Mistborn: The Final Empire', author_name: 'Brandon Sanderson' },
  { title: 'The Well of Ascension', author_name: 'Brandon Sanderson' },
  ...
  { title: 'Wind and Truth', author_name: 'Brandon Sanderson' }
]

The condition could be written more simply as where author.isAlive here too.

An illustration of the opposite might look like this (again, using terser condition syntax, instead of, say, where isAlive = false):

> await cds.ql`
  SELECT from ${Authors} { name, books.title }
  where isAlive = false
  `
[
  { name: 'Emily Brontë', books_title: 'Wuthering Heights' },
  { name: 'Charlotte Brontë', books_title: 'Jane Eyre' },
  { name: 'Edgar Allen Poe', books_title: 'Eleonora' },
  { name: 'Edgar Allen Poe', books_title: 'The Raven' },
  { name: 'Richard Carpenter', books_title: 'Catweazle' },
  { name: 'J. R. R. Tolkien', books_title: 'Beren and Lúthien' },
  { name: 'J. R. R. Tolkien', books_title: 'The Children of Húrin' },
  ...
  { name: 'J. R. R. Tolkien', books_title: 'Unfinished Tales' }
]

Here also the condition could be written more simply as where not isAlive.

Target references

33:11 There was a brief discussion about the "Books" reference in the previous query, which looked like this:

SELECT from ${Books}

This is the more precise approach to specifying the target of the query - the entity represented by (contained in) the Books variable injected into the cds REPL session, which resolves thus:

> Books.name
sap.capire.bookshop.Books

When we use a literal value instead, like this:

SELECT from Books

then CAP can often resolve the reference, but caution must be used here in case there are other entities, in different scopes, but with the same name; in this case using the template string interpolation (${ ... }) will allow us to be specific.

Query details and the phased translation to native SQL

35:05 At this point Patrice digs in a little deeper to the query. First, from the CQL (which was then modified at 36:25, so that the from reference is to sap.capire.bookshop.Books rather than just Books), we see the CQN representation which includes an expression notation for the isAlive check:

> q = cds.ql`
  SELECT from ${Books} { title, author.name }
  where author.isAlive = false
  `
cds.ql {
  SELECT: {
    from: { ref: [ 'sap.capire.bookshop.Books' ] },
    columns: [ { ref: [ 'title' ] }, { ref: [ 'author', 'name' ] } ],
    where: [ { ref: [ 'author', 'isAlive' ] }, '=', { val: false } ]
  }
}

Using the toSQL() method on the query object, we see the SQL¹ and the injectable values:

> q.toSQL()
{
  sql: `SELECT json_insert('{}','$."title"',title,'$."author_name"',author_name) as _json_ FROM (SELECT Books.title,author.name as author_name FROM sap_capire_bookshop_Books as Books left JOIN sap_capire_bookshop_Authors as author ON author.ID = Books.author_ID WHERE (case when author.dateOfDeath is null then ? else ? end) = ?)`,
  values: [ 1, 0, 0 ]
}

And the SQL, when formatted nicely, looks like this:

SELECT
  json_insert(
    '{}', '$."title"', title, '$."author_name"', author_name
  ) as _json_
FROM
  (
    SELECT
      Books.title,
      author.name as author_name
    FROM sap_capire_bookshop_Books as Books
      left JOIN sap_capire_bookshop_Authors as author
      ON author.ID = Books.author_ID
    WHERE
      (
        case
          when author.dateOfDeath is null then ?
          else ?
        end
      ) = ?
  )

38:30 We've jumped from CQL almost directly to the (SQLite dialect² of) SQL here, but Patrice now explains the multi-step process here.

Between the CQL (and CXL), and its machine-readable CQN (and CXN) equivalents, and the ultimate persistence-layer-specific SQL, there's an intermediate "normalised" format. This is often referred to as "CAP-style SQL", or the "SQL variant of CQL". Using the forSQL() method on the query object (as opposed to the toSQL() we used just now), we can get this intermediate format:

> q.forSQL()
cds.ql {
  SELECT: {
    from: {
      join: 'left',
      args: [
        { ref: [ 'sap.capire.bookshop.Books' ], as: 'Books' },
        { ref: [ 'sap.capire.bookshop.Authors' ], as: 'author' }
      ],
      on: [
        { ref: [ 'author', 'ID' ] },
        '=',
        { ref: [ 'Books', 'author_ID' ] }
      ]
    },
    columns: [
      { ref: [ 'Books', 'title' ] },
      { ref: [ 'author', 'name' ], as: 'author_name' }
    ],
    where: [
      {
        xpr: [
          'case',
          'when',
          { ref: [ 'author', 'dateOfDeath' ] },
          'is',
          'null',
          'then',
          { val: true },
          'else',
          { val: false },
          'end'
        ]
      },
      '=',
      { val: false }
    ]
  }
}

As we can see, we can recognise both CXL (CXN) style expressions, as well as SQL-style constructions such as JOINs. At this point this is is still in a form that is not SQL database system specific (i.e. not SQLite, Postgres or HANA specific SQL).

And as opposed to this neutral "normalised" format from forSQL(), we get the database-specific dialect with toSQL().

A question on WITH ASSOCIATIONS and the previous db adapters

41:20 At this point ArtlessSoul asks a question to which Patrice responds by explaining the difference between how path expressions were handled (translated to SQL) in the now-legacy database services which were in play before the current major CAP version.

It had not been possible at the time to transform all path expressions to the required JOINs that would be needed to represent them at a SQL level. One way to address this at the time, specifically for HANA, had been to push down such associations to the database, which supported them with a WITH ASSOCIATIONS native feature (see the Native Associations section of the SAP HANA topic in Capire).

While HANA would be the typical target database runtime for production, it had thus not been previously possible to test associations in development, i.e. outside the HANA context.

But thanks to the new database adapters with the current major release, all associations are possible for all supported databases due to improvements in how they're managed and translated by the CAP runtime, and therefore a much higher development and testing confidence can be achieved.

Exploring function expressions

45:01 Patrice starts to round off this episode with a nice example to illustrate function expressions. The example is another calculated element on the Authors entity, for the author's age:

age : Integer = years_between(dateOfBirth, coalesce(dateOfDeath, current_date));

46:38 But before continuing, he connects and deploys to the HANA Cloud service he has already set up in his account, following the same connection and deployment to HANA procedure from part 2, and then at 48:20 he starts the cds REPL using the hybrid profile, also following the same procedure from part 2 (see the Going hybrid section of part 2's notes for details).

49:10 At this point we take a look at what happens under the hood -- in HANA -- for this new age element, step by step.

First, step 1, from CQL to CQN and the query object:

> q = cds.ql`SELECT from ${Authors} { name, age }`
cds.ql {
  SELECT: {
    from: { ref: [ 'sap.capire.bookshop.Authors' ] },
    columns: [ { ref: [ 'name' ] }, { ref: [ 'age' ] } ]
  }
}

Next, step 2, taking a look at the intermediate "normalised" version of the query, using forSQL():

> q.forSQL()
cds.ql {
  SELECT: {
    from: { ref: [ 'sap.capire.bookshop.Authors' ], as: 'Authors' },
    columns: [
      { ref: [ 'Authors', 'name' ] },
      {
        args: [
          { ref: [ 'Authors', 'dateOfBirth' ] },
          {
            func: 'coalesce',
            args: [
              { ref: [ 'Authors', 'dateOfDeath' ] },
              { func: 'current_date' }
            ]
          }
        ],
        func: 'years_between',
        as: 'age'
      }
    ]
  }
}

From a column perspective, there are two:

a reference to the name element in sap.capire.bookshop.Authors
the evaluation of a function years_between, the arguments to which are
- a reference to the dateOfBirth element
- the evaluation of another function coalesce, the arguments to which are
  - a reference to the dateOfDeath element
  - the evaluation of yet another function current_date

Running this query gives us what we expect:

> await q
[
  { age: 30, name: 'Emily Brontë' },
  { age: 36, name: 'Charlotte Brontë' },
  { age: 40, name: 'Edgar Allen Poe' },
  { age: 82, name: 'Richard Carpenter' },
  { age: 50, name: 'Brandon Sanderson' },
  { age: 81, name: 'J. R. R. Tolkien' }
]

51:05 After adding the isAlive element back into the query, Patrice now takes a look at the actual SQL resolved for this query ... in the context of the hybrid profile, which means in the context of a HANA database:

> cds.ql`SELECT from ${Authors} {name, age, isAlive }`.toSQL().sql

This results in SQL that, when formatted, looks like this:

SELECT
  Authors.name as "name",
  years_between (
    Authors.dateOfBirth,
    coalesce(Authors.dateOfDeath, current_utcdate)
  ) as "age",
  case
    when Authors.dateOfDeath is null then true
    else false
  end as "isAlive"
FROM
  sap_capire_bookshop_Authors as Authors

There's pretty much a 1-to-1 correlation between the CDL we have used to define our Authors entity, and the HANA-flavoured SQL here.

Portable functions

And that's because HANA implements the years_between function natively. Unlike SQLite, for which more heavy lifting is done by the compiler team, to essentially provide us with "a set of portable functions (and also operators) which are automatically translated for us to the best-possible database-specific native SQL equivalents". That quote is from Capire's CAP-Level Database Support topic which is well worth a read.

52:47 To illustrate this, Patrice starts a new cds REPL session in the context of the default (development) profile which implies SQLite:

cds repl --run .

This time, the same query (SELECT from ${Authors} { name, age }) results in the same CQN of course, as well as the same normalised CAP-style SQL (via forSQL()) as before, but the resulting database-specific SQL(via toSQL().sql) is quite different³:

SELECT
  json_insert ('{}', '$."name"', name, '$."age"', age) as _json_
FROM
  (
    SELECT
      "Authors".name,
      floor(
        (
          (
            (
              cast(
                strftime ('%Y', coalesce("Authors".dateOfDeath, current_date)) as Integer
              ) - cast(strftime ('%Y', "Authors".dateOfBirth) as Integer)
            ) * 12
          ) + (
            cast(
              strftime ('%m', coalesce("Authors".dateOfDeath, current_date)) as Integer
            ) - cast(strftime ('%m', "Authors".dateOfBirth) as Integer)
          ) + (
            (
              case
                when (
                  cast(
                    strftime ('%Y%m', coalesce("Authors".dateOfDeath, current_date)) as Integer
                  ) < cast(strftime ('%Y%m', "Authors".dateOfBirth) as Integer)
                ) then (
                  cast(
                    strftime (
                      '%d%H%M%S%f0000',
                      coalesce("Authors".dateOfDeath, current_date)
                    ) as Integer
                  ) > cast(
                    strftime ('%d%H%M%S%f0000', "Authors".dateOfBirth) as Integer
                  )
                )
                else (
                  cast(
                    strftime (
                      '%d%H%M%S%f0000',
                      coalesce("Authors".dateOfDeath, current_date)
                    ) as Integer
                  ) < cast(
                    strftime ('%d%H%M%S%f0000', "Authors".dateOfBirth) as Integer
                  )
                ) * -1
              end
            )
          )
        ) / 12
      ) as age
    FROM
      sap_capire_bookshop_Authors as "Authors"
  )

As Patrice reminds us, this is another example of AXI001 What not how in action.

56:45 To wrap this section up, Patrice adds back in the isAlive element to the query before executing it, to show that this works too, albeit resulting in a different⁴ representation of true and false, as SQLite has no Boolean data type:

> await cds.ql`SELECT from ${Authors} {name, age, isAlive }`
[
  { name: 'Emily Brontë', age: 30, isAlive: 0 },
  { name: 'Charlotte Brontë', age: 36, isAlive: 0 },
  { name: 'Edgar Allen Poe', age: 40, isAlive: 0 },
  { name: 'Richard Carpenter', age: 82, isAlive: 0 },
  { name: 'Brandon Sanderson', age: 50, isAlive: 1 },
  { name: 'J. R. R. Tolkien', age: 81, isAlive: 0 }
]

Support in CDL as well as CQL

58:25 Wrapping up, Patrice makes the very good point that this portability afforded by CAP to support different database runtimes is not only at the "runtime" level, i.e. in queries (as we've seen in these examples), but also at the persistence, or data definition level (in particular for views). To illustrate this, Patrice shows us that the view DDL is constructed appropriately, first in the production profile context, i.e. for HANA:

; cds build --production
building project with {
  versions: { cds: '9.8.0', compiler: '6.8.0', dk: '9.4.3' },
  target: 'gen',
  tasks: [
    { src: 'db', for: 'hana', options: { model: [ 'db', 'srv', '@sap/cds/srv/outbox' ] } },
    { src: 'srv', for: 'nodejs', options: { model: [ 'db', 'srv', '@sap/cds/srv/outbox' ] } }
  ]
}
done > wrote output to:
   gen/db/package.json
   gen/db/src/.hdiconfig
   gen/db/src/gen/.hdiconfig
   gen/db/src/gen/.hdinamespace
   gen/db/src/gen/AdminService.Authors.hdbview
   gen/db/src/gen/AdminService.Books.hdbview
   ... more. Run with DEBUG=build to show all files.

build completed in 2438 ms

The DDL for the Authors projection in the AdminService (in gen/db/src/gen/AdminService.Authors.hdbview) looks like this, where the HANA native years_between function is available:

VIEW AdminService_Authors AS SELECT
  Authors_0.createdAt,
  Authors_0.createdBy,
  Authors_0.modifiedAt,
  Authors_0.modifiedBy,
  Authors_0.ID,
  Authors_0.name,
  Authors_0.address_ID,
  Authors_0.academicTitle,
  Authors_0.dateOfBirth,
  Authors_0.dateOfDeath,
  Authors_0.placeOfBirth,
  Authors_0.placeOfDeath,
  CASE WHEN Authors_0.dateOfDeath IS NULL THEN TRUE ELSE FALSE END AS isAlive,
  years_between(Authors_0.dateOfBirth, coalesce(Authors_0.dateOfDeath, current_date)) AS age
FROM sap_capire_bookshop_Authors AS Authors_0

In contrast, when, like earlier, we compile the AdminService definitions in the default development profile context i.e. for SQLite:

cds compile -2 sql srv/admin-service.cds

we get this for the Authors projection:

CREATE VIEW AdminService_Authors AS
SELECT
  Authors_0.createdAt,
  Authors_0.createdBy,
  Authors_0.modifiedAt,
  Authors_0.modifiedBy,
  Authors_0.ID,
  Authors_0.name,
  Authors_0.address_ID,
  Authors_0.academicTitle,
  Authors_0.dateOfBirth,
  Authors_0.dateOfDeath,
  Authors_0.placeOfBirth,
  Authors_0.placeOfDeath,
  CASE
    WHEN Authors_0.dateOfDeath IS NULL THEN TRUE
    ELSE FALSE
  END AS isAlive,
  floor(
    (
      (
        (
          (
            CAST(
              strftime (
                '%Y',
                coalesce(Authors_0.dateOfDeath, current_date)
              ) AS Integer
            ) - CAST(strftime ('%Y', Authors_0.dateOfBirth) AS Integer)
          ) * 12
        ) + (
          CAST(
            strftime (
              '%m',
              coalesce(Authors_0.dateOfDeath, current_date)
            ) AS Integer
          ) - CAST(strftime ('%m', Authors_0.dateOfBirth) AS Integer)
        ) + (
          CASE /* For backward intervals: if the composite (day + time) of y is greater than x, add 1. */
            WHEN CAST(
              strftime (
                '%Y%m',
                coalesce(Authors_0.dateOfDeath, current_date)
              ) AS Integer
            ) < CAST(
              strftime ('%Y%m', Authors_0.dateOfBirth) AS Integer
            ) THEN (
              CAST(
                strftime (
                  '%d%H%M%S%f0000',
                  coalesce(Authors_0.dateOfDeath, current_date)
                ) AS Integer
              ) > CAST(
                strftime ('%d%H%M%S%f0000', Authors_0.dateOfBirth) AS Integer
              )
            ) /* For forward intervals: if the composite of y is less than x, subtract 1. */
            ELSE (
              CAST(
                strftime (
                  '%d%H%M%S%f0000',
                  coalesce(Authors_0.dateOfDeath, current_date)
                ) AS Integer
              ) < CAST(
                strftime ('%d%H%M%S%f0000', Authors_0.dateOfBirth) AS Integer
              )
            ) * -1
          END
        )
      )
    ) / 12
  ) AS age
FROM
  sap_capire_bookshop_Authors AS Authors_0;

Phew!

Further info

Footnotes

This is what was show on Patrice's screen; what you will likely get instead is SQL that is almost the same, except that a technical alias name is used for the FROM target, i.e. you will likely see "$B" instead of Books, like this:

SELECT
  json_insert(
    '{}', '$."title"', title, '$."author_name"', author_name
  ) as _json_
FROM
  (
    SELECT
      "$B".title,
      author.name as author_name
    FROM sap_capire_bookshop_Books as "$B"
      left JOIN sap_capire_bookshop_Authors as author
      ON author.ID = "$B".author_ID
    WHERE
      (
        case
          when author.dateOfDeath is null then ?
          else ?
        end
      ) = ?
  )

Technical aliases are discussed in a later episode in this series.

The SQL that we see here is the SQLite dialect as we're running in development mode by default.
Notice that the coalesce function is available natively not only in HANA but also in SQLite.

When using the hybrid profile and connected to HANA, this is how the query runs:

> await cds.ql`SELECT from ${Authors} {name, age, isAlive }`
[
  { age: 30, isAlive: false, name: 'Emily Brontë' },
  { age: 36, isAlive: false, name: 'Charlotte Brontë' },
  { age: 40, isAlive: false, name: 'Edgar Allen Poe' },
  { age: 82, isAlive: false, name: 'Richard Carpenter' },
  { age: 50, isAlive: true, name: 'Brandon Sanderson' },
  { age: 81, isAlive: false, name: 'J. R. R. Tolkien' }
]

CDS expressions in CAP - notes on Part 2

2026-03-09T00:00:00Z

See the series post for an overview of all the episodes.

Introduction

00:00 A rather lengthy introduction and recap (sorry Patrice!).

The case-when-then-else-end expression

09:30 Patrice reviews and re-explains the CASE ... WHEN ... THEN ... ELSE ... END expression in the @assert annotation, noting that if neither of the conditions match then the expression evaluates to null:

annotate AdminService.Books:stock with @assert: (
  case
    when stock < 0 then 'stock must not be negative'
    when stock > 1000 then 'stock exceeds maximum limit of 1000'
  end
);

13:15 Starting the cds REPL with DEBUG=sql to see what happens under the hood:

DEBUG=sql cds r --run .

Immediately we see many SQL statements being executed due to the automatic deployment to the in-memory SQLite database that's used:

DROP VIEW IF EXISTS ...
DROP TABLE IF EXISTS ...
CREATE TABLE ...
INSERT INTO ...

15:15 Patrice explains what we missed in the previous part - the db service does not have the hook for the assertion, it's the AdminService that does, and that's because it lives not in the DB service but in the Application Service.

22:00 At this point Patrice explains what actually happened, by following the detail emitted in the SQL debug output. There's also a dedicated blog post on this detail: Constraints, expressions and axioms in action.

The key SQL statement that we focused on was this one:

SELECT
  Books.ID,
  case
      when Books.stock < ? then ?
      when Books.stock > ? then >
  end as "@assert:stock"
FROM AdminService_Books as Books
WHERE (Books.ID) in ((?), (?))

This is executed within the transaction, straight after the record insertion, and, in the @assert:stock alias, surfaces either nothing or an error string (from the expression in the @assert annotation above); the selection is restricted to the two records specifically just inserted. If something (i.e. an error string) is surfaced from this, then the entire transaction is aborted and rolled back.

31:20 While some checks are done implicitly in the "before" phase, such as in the built-in validate_input here:

> AdminService.handlers
EventHandlers {
  _initial: [
    {
      before: '*',
      handler: [Function: check_service_level_restrictions]
    },
    { before: '*', handler: [Function: check_auth_privileges] },
    { before: '*', handler: [Function: check_readonly] },
    { before: '*', handler: [Function: check_insertonly] },
    { before: '*', handler: [Function: check_odata_constraints] },
    { before: '*', handler: [Function: check_autoexposed] },
    { before: '*', handler: [AsyncFunction: enforce_auth] },
    { before: 'READ', handler: [Function: restrict_expand] },
    { before: 'CREATE', handler: [AsyncFunction: validate_input] },
    { before: 'UPDATE', handler: [AsyncFunction: validate_input] },
    { before: 'NEW', handler: [AsyncFunction: validate_input] },
    { before: 'READ', handler: [Function: handle_paging] },
    { before: 'READ', handler: [Function: handle_sorting] }
  ],
  ...
}

... such declarative constraints are checked in an "after" phase handler - see the notes to part 3 of this series for a clarification.

A question on reusable expressions

31:30 At this point we start to look at one of the two questions that were asked at the very end of part 1 by Ben: "Would it be possible to build an expression and then re-use it for different assertions?".

Patrice's answer was based on CAP's embrace of aspect orientation, specifically at the "db" level, in db/schema.cds. First, an aspect defined thus:

aspect ConstrainedTitle {
  @assert: (
    case
      when length(title) < 2 then 'title must be at least 2 characters long'
    end
  )
  title : String @mandatory;
}

and employed like this (where the basic Books entity definition does not now have its own title element):

entity Books : managed, ConstrainedTitle {
  key ID    : Integer;
  descr     : localized String(1111);
  ...
}

Looking at the CSN, we see that this constraint is included in the AdminService as we'd expect:

; echo 'AdminService.entities.Books.elements.title' | cds r -r .
...
String {
  '@assert': {
    '=': "case when length(title) < 2 then 'title must be at least 2 characters long' end",
    xpr: [
      'case',
      'when',
      {
        func: 'length',
        args: [ { ref: [ 'title' ] } ]
      },
      '<',
      { val: 2 },
      'then',
      { val: 'title must be at least 2 characters long' },
      'end'
    ]
  },
  '@mandatory': true,
  type: 'cds.String',
  '@Common.FieldControl': { '#': 'Mandatory' }
}

36:56 This constraint annotation is propagated up to the "service" level where there's a projection on the Books entity. Patrice also points out here something that we might not expect, but happens anyway thanks to the compiler ... even if we change the name of the element, say from title to mytitle in the projection, the expression is modified accordingly!

To see this in action, here's a temporary modification to the Books projection in the AdminService, renaming the title element to mytitle:

using {sap.capire.bookshop as my} from '../db/schema';

service AdminService {
  entity Books   as
    projection on my.Books {
      *,
      title as mytitle
    }
    excluding { title }

  ...
}

The change is effected not only at the element level, but within the expression too:

; echo 'AdminService.entities.Books.elements.mytitle' | cds r -r .
String {
  '@assert': {
    '=': true,
    xpr: [
      'case',
      'when',
      {
        func: 'length',
        args: [ { ref: [ 'mytitle' ] } ]
      },                     ^
      '<',                   |
      { val: 2 },            +--- modified within the expression
      'then',
      { val: 'title must be at least 2 characters long' },
      'end'
    ]
  },
  '@mandatory': true,
  type: 'cds.String',
  '@Common.FieldControl': { '#': 'Mandatory' }
}

This constraint works as expected, as we can see from this cds REPL session:

> { Books } = AdminService.entities
[object Function]
> insert = INSERT.into(Books).entries({ID:567,author_ID:180,title:"X",stock:1})
cds.ql {
  INSERT: {
    into: { ref: [ 'AdminService.Books' ] },
    entries: [ { ID: 567, author_ID: 180, title: 'X', stock: 1 } ]
  }
}
> await AdminService.run(insert)
Uncaught:
{
  status: 400,
  code: 'ASSERT',
  target: 'title',
  message: 'title must be at least 2 characters long'
}

Connection and deployment to HANA

39:52 To show that this entire approach is portable across, and abstract from the underlying database systms supported by CAP, Patrice proceeds at this point to show the same thing but with HANA.

First, adding database support for HANA via the facet:

; cds add hana

Adding facet: hana

Successfully added features to your project

Amongst other things (such as creating db/undeploy.json), this adds @cap-js/hana as a dependency:

--- a/package.json
+++ b/package.json
@@ -3,6 +3,7 @@
   "version": "1.0.0",
   "description": "A simple CAP project.",
   "dependencies": {
+    "@cap-js/hana": "^2",
     "@sap/cds": "^9",
     "express": "^4"
   },

Which, when installed (with npm install), adds HANA specific DB parameters to the production profile, which we can see with:

; cds env requires.db --profile production
{
  impl: '@cap-js/hana',
  data: [ 'db/data', 'db/csv' ],
  pool: {
    min: 0,
    max: 10,
    acquireTimeoutMillis: 1000,
    idleTimeoutMillis: 60000,
    evictionRunIntervalMillis: 100000,
    numTestsPerEvictionRun: 10,
    testOnBorrow: true,
    fifo: false
  },
  kind: 'hana'
}

42:49 Now everything is ready for a deployment to HANA, which Patrice does at this point:

; cds deploy --to hana
building project with {
  versions: { cds: '9.8.0', compiler: '6.8.0', dk: '9.7.2' },
  target: 'gen',
  tasks: [
    { for: 'hana', src: 'db', options: { model: [ 'db', 'srv', '@sap/cds/srv/outbox' ] } }
  ]
}
done > wrote output to:
   gen/db/package.json
   gen/db/src/gen/.hdiconfig
   gen/db/src/gen/.hdinamespace
   gen/db/src/gen/AdminService.Authors.hdbview
   gen/db/src/gen/AdminService.Books.hdbview
   ...

build completed in 332 ms

using container cxl-bookshop-db
creating service cxl-bookshop-db - please be patient...
creating service key cxl-bookshop-db-key - please be patient...
starting deployment to SAP HANA ...
deploying to HANA from /work/gh/github.com/patricebender/cxl-bookshop/gen/db/
HDI deployer path: /home/dj/.npm-packages/lib/node_modules/@sap/cds-dk/node_modules/@sap/hdi-deploy/library.js
HDI deployer version: 5.6.1
VCAP_SERVICES:

{
  "hana": [
    {
      "name": "cxl-bookshop-db",
      "label": "hana",
      "credentials": {
        "schema": "4DCBA6B626C745E29947D848D3CCEE38",
        "database_id": "442316b0-3ec9-469c-b3ab-2d333c2ecf8f",
        "user": "4DCBA6B626C745E29947D848D3CCEE38_BROA7XLTTVAJ193W0CETRKJF2_RT",
        "hdi_user": "4DCBA6B626C745E29947D848D3CCEE38_BROA7XLTTVAJ193W0CETRKJF2_DT"
      },
      "tags": [
        "hana"
      ]
    }
  ]
}

...

Deployment started ...

   ... (lots of log output deleted)

   Processing work list... ok (2s 70ms)
   Finalizing... ok (0s 126ms)
   Make succeeded (0 warnings): 51 files deployed (effective 58), 0 files undeployed (effective 0), 0 dependent files redeployed
  Making... ok (4s 497ms)
  Enabling table replication for the container schema "4DCBA6B626C745E29947D848D3CCEE38"...
  Enabling table replication for the container schema "4DCBA6B626C745E29947D848D3CCEE38"... ok (0s 17ms)
 Starting make in the container "4DCBA6B626C745E29947D848D3CCEE38" with 51 files to deploy, 0 files to undeploy... ok (4s 586ms)

Deploying to the container "4DCBA6B626C745E29947D848D3CCEE38"... ok (10s 853ms)
...
Deployment ended ...
...
retrieving data from Cloud Foundry...
binding db to Cloud Foundry managed service cxl-bookshop-db:cxl-bookshop-db-key with kind hana
saving bindings to .cdsrc-private.json in profile hybrid
successfully finished deployment

Going hybrid

Right at the end we can see the bindings saved in a project-local .cdsrc-private.json file, under the "hybrid" profile name (see the Hybrid Testing topic in Capire); the binding information looks similar to this:

{
  "requires": {
    "[hybrid]": {
      "db": {
        "binding": {
          "type": "cf",
          "apiEndpoint": "https://api.cf.us10-001.hana.ondemand.com",
          "org": "38b4ec42trial",
          "space": "dev",
          "instance": "cxl-bookshop-db",
          "key": "cxl-bookshop-db-key"
        },
        "kind": "hana",
        "vcap": {
          "name": "db"
        }
      }
    }
  }
}

This binding information is then used to start a cds REPL session ... but in the context of the remote HANA database system¹:

; cds bind --exec --profile hybrid -- cds repl --run .
resolving cloud service bindings...
bound db to cf managed service cxl-bookshop-db:cxl-bookshop-db-key
Welcome to cds repl v 9.8.0
[cds] - using bindings from: { registry: '~/.cds-services.json' }
[cds] - loaded model from 4 file(s):

  srv/cat-service.cds
  srv/admin-service.cds
  db/schema.cds
  node_modules/@sap/cds/common.cds

[cds] - connect to db > hana {
  database_id: '442316b0-3ec9-469c-b3ab-2d333c2ecf8f',
  host: '442316b0-3ec9-469c-b3ab-2d333c2ecf8f.hna1.prod-us10.hanacloud.ondemand.com',
  port: '443',
  driver: 'com.sap.db.jdbc.Driver',
  url: 'jdbc:sap://442316b0-3ec9-469c-b3ab-2d333c2ecf8f.hna1.prod-us10.hanacloud.ondemand.com:443?encrypt=true&validateCertificate=true&currentschema=4DCBA6B626C745E29947D848D3CCEE38',
  schema: '4DCBA6B626C745E29947D848D3CCEE38',
  certificate: '...',
  hdi_user: '4DCBA6B626C745E29947D848D3CCEE38_BROA7XLTTVAJ193W0CETRKJF2_DT',
  hdi_password: '...',
  user: '4DCBA6B626C745E29947D848D3CCEE38_BROA7XLTTVAJ193W0CETRKJF2_RT',
  password: '...'
}
[cds] - using auth strategy { kind: 'mocked' }
[cds] - serving AdminService {
  at: [ '/odata/v4/admin' ],
  decl: 'srv/admin-service.cds:3'
}
[cds] - serving CatalogService {
  at: [ '/odata/v4/catalog' ],
  decl: 'srv/cat-service.cds:3'
}
[cds] - server listening on { url: 'http://localhost:45699' }
[cds] - server v9.8.0 launched in 555 ms
[cds] - [ terminate with ^C ]


------------------------------------------------------------------------
Following variables are made available in your repl's global context:

from cds.entities: {
  Books,
  Authors,
  Addresses,
  Cities,
  Genres,
  Orders,
  OrderItems,
}

from cds.services: {
  db,
  AdminService,
  CatalogService,
}

Simply type e.g. CatalogService in the prompt to use the respective objects.

This setup is now complete and we can see that the constraint works here on HANA just the same, too:

> { Books } = AdminService.entities
[object Function]
> await AdminService.run(INSERT.into(Books).entries({ ID:567, author_ID:180, title:"X", stock:1 }))
Uncaught:
{
  status: 400,
  code: 'ASSERT',
  target: 'title',
  message: 'title must be at least 2 characters long'
}

A question on iteration and aggregation in expressions

46:27 Next, Patrice turns to the second of the two questions asked in the previous episode, this time one from Stubbs: "Can we iterate over composition items with expressions? Example calculate order total amount using line item amounts?". I've written about the answer to this question in great detail in the post Path expressions, nested projections, aggregations and expressions in queries with CQL and CXL in CAP, so won't dwell any more on it in these notes here.

Exploring with cds.parse.expr

54:32 Rounding off this episode, Patrice explores some of the brand new documentation in Capire for CXL, and emphasises how easy it is to try things out in the cds REPL.

Starting out as simple as possible with just a literal:

> cds.parse.expr` 42 `
{ val: 42 }

Then with one of the unary operators not:

> cds.parse.expr` not foo `
{
  xpr: [ 'not', { ref: [ 'foo' ] } ]
}

And with the binary operator +, producing an expression made up of a literal ({ val: 1 }) followed by the binary operator '+' followed by another literal ({ val: 1}):

> cds.parse.expr` 1 + 1 `
{ xpr: [ { val: 1 }, '+', { val: 1 } ] }

With growing confidence we can now construct larger expressions, this time a predicate (based on the > binary operator):

> cds.parse.expr` (1 + 1) > (1 + 2) `
{
  xpr: [
    { xpr: [ { val: 1 }, '+', { val: 1 } ] },
    '>',
    { xpr: [ { val: 1 }, '+', { val: 2 } ] }
  ]
}

Definitely worth trying a few out yourself!

Further info

Patrice also has some great notes for this part.

Footnotes

The "hybrid" profile is used as default by cds bind so we can omit the --profile hybrid here:
```
cds bind --exec -- cds repl --run .
```

CDS expressions in CAP - notes on Part 1

2026-03-05T00:00:00Z

See the series post for an overview of all the episodes.

Introduction

00:00 Introductions.

05:00 A first look at expressions in the context of the new Declarative Constraints feature.

The family of CDS languages

06:00 An overview of CDS and the languages within the CDS family (CDL & CSN, CQL & CQN and CXL & CXN).

07:05 A first look at the "cxl-bookshop" project, specifically the entities and service definition. Patrice compiled (with cds compile db/schema.cds) the schema level definitions to show the relationship between the human-first CDL (Conceptual Definition Language) and the corresponding machine-first CSN (Core Schema Notation).

11:38 Next in the CDS language family overview is the human-first CQL (CDS Query Language) and corresponding machine-first equivalent CQN (CDS Query Notation). An example was given in the context of a cds REPL session (started with cds repl --run .). The example uses a postfix projection to specify the desired column (element), instead of the more SQL-like SELECT title from Books (the resulting query is the same):

> q = cds.ql` SELECT from Books { title }`
cds.ql {
  SELECT: {
    from: { ref: [ 'Books' ] },
    columns: [ { ref: [ 'title' ] } ]
  }
}

This shows the CQL and the corresponding CQN.

19:58 The final language pair, the human-first CXL (CDS Expression Language) and the corresponding machine-first CXN (CDS Expression Notation) (at this point the CXL documentation in Capire was still under construction, but exists now).

A first look at expressions

27:23 Looking at very simple expressions, trying out the (context-free) expression parser cds.parse.expr to parse a CXL expression to the internal CXN representation:

> cds.parse.expr` 1 `
{ val: 1 }

Here's one with a binary operator (+), taking two operands:

> cds.parse.expr` 1 + 1 `
{ xpr: [ { val: 1 }, '+', { val: 1 } ] }

29:01 An example in the context of declarative constraints, where validation is added to a service entity (which makes sense as they act as functional facets to the underlying persistence layer), in particular, to the stock element of the Books entity:

using {sap.capire.bookshop as my} from '../db/schema';

service AdminService {
  entity Books   as projection on my.Books;
  entity Authors as projection on my.Authors;
}

annotate AdminService.Books:stock with @assert: (
  case
    when stock < 0 then 'stock must not be negative'
    when stock > 1000 then 'stock exceeds maximum limit of 1000'
  end
);

Note that here we have:

the annotate directive (CDL)
the @assert annotation (also CDL)
the case ... when ... then ... end expression (CXL)

There was a brief discussion on how expressions in annotations should be enclosed in parentheses (as used in this annotation example), which also allows full compiler support and also code completion.

Exercising the constraint in the cds REPL

34:58 First, an overview of the entities:

> Object.keys(AdminService.entities)
[
  'Books',
  'Authors',
  'Genres',
  'Genres.texts',
  'Currencies',
  'Currencies.texts',
  'Books.texts'
]

and then some destructuring to get the Books entity from the AdminService:

{ Books } = AdminService.entities
[object Function]

Next, the construction of an INSERT query via the fluent API (the statement is split across multiple lines for readability; use the REPL's .editor command to paste in these multiple lines if you're playing along):

> insert = INSERT
.into(Books)
.entries(
  { ID:567, author_ID: 180, title: 'Foo', stock: -1 },
  { ID: 568, author_ID: 180, title: 'Foo 2', stock: 1001 }
)

which gives:

cds.ql {
  INSERT: {
    into: { ref: [ 'AdminService.Books' ] },
    entries: [
      { ID: 567, author_ID: 180, title: 'Foo', stock: -1 },
      { ID: 568, author_ID: 180, title: 'Foo 2', stock: 1001 }
    ]
  }
}

At this point, this query was awaited, with an unexpected result. That's because using await on a query object will by default invoke db.run(), sending that query object as an argument (i.e. db.run(insert) here). However, this db service is the "wrong" one - it's the persistence-layer database service, rather than the AdminService at which level we have our declarative constraint. This is why the insertion of the two new book records was successful (at 39:50).

Sending this to the AdminService to be executed:

> AdminService.run(insert)

shows us the result of the constraint:

> Uncaught [Error: MULTIPLE_ERRORS] {
  details: [
    {
      status: 400,
      code: 'ASSERT',
      target: 'stock',
      numericSeverity: 4,
      '@Common.numericSeverity': 4,
      message: 'stock must not be negative'
    },
    {
      status: 400,
      code: 'ASSERT',
      target: 'stock',
      numericSeverity: 4,
      '@Common.numericSeverity': 4,
      message: 'stock exceeds maximum limit of 1000'
    }
  ]
}

Comparing entity definitions at different layers

46:25 A brief comparison between the Books definition at the persistence ("db") level from cds.entities and the Books definition in the AdminService at the service ("srv") level. This is effectively the difference between A and B where:

using {sap.capire.bookshop as my} from '../db/schema';

service AdminService {
  entity Books as projection on my.Books;
           ^                       ^
           |                       |
           B                       A

A my.Books is from sap.capire.bookshop in db/schema
B Books is the entity defined in AdminService as a projection on my.Books in A

Comparing the artifacts and their CSN shows differences that all make sense (each of these comparisons shows A first, then B):

the artifacts' name properties (Books.name) are different: sap.capire.bookshop.Books vs AdminService.Books

reflecting this, the scope name prefixes are different in each reference, with A being sap.capire.bookshop and B being AdminService, for example:

author: Association {
  ...
  target: 'sap.capire.bookshop.Authors'
}

author: Association {
  ...
  target: 'AdminService.Authors'
}

each association in the service level CSN has an additional annotation @Common.ValueList.viaAssociation, for example (although we didn't dwell on this at all):

  genre: Association {
    type: 'cds.Association',
    target: 'AdminService.Genres',
    keys: [ { ref: [ 'ID' ], '$generatedFieldName': 'genre_ID' } ],
    '@Common.ValueList.viaAssociation': { '=': 'genre' }
  }

as pointed out in the conversation, the service level CSN has an extra projection property, which makes sense given how this is defined:
```
projection: { from: { ref: [ 'sap.capire.bookshop.Books' ] } }
```

And perhaps most significantly:

while the stock element at the persistence level is defined like this:

stock: Integer { type: 'cds.Integer' }

the projected version, thanks to the annotation applied specifically to AdminService.Books:stock, is defined like this (where can see the expression in CXN form):

stock: Integer {
  '@assert': {
    '=': "case when stock < 0 then 'stock must not be negative' when stock > 1000 then 'stock exceeds maximum limit of 1000' end",
    xpr: [
      'case',
      'when',
      { ref: [ 'stock' ] },
      '<',
      { val: 0 },
      'then',
      { val: 'stock must not be negative' },
      'when',
      { ref: [ 'stock' ] },
      '>',
      { val: 1000 },
      'then',
      { val: 'stock exceeds maximum limit of 1000' },
      'end'
    ]
  },
  type: 'cds.Integer'
}

57:44 Patrice describes the relationship between CXL, CQL and CDL in terms of "passenger" and "vehicle", in that CXL expressions are "passengers" carried in the "vehicles" of CQL queries and / or CDL definitions (such as annotations and element definitions); Patrice illustrates this further by showing what a simple expression in CXL is parsed to:

> cds.parse.expr` author.books.genre.name `
{ ref: [ 'author', 'books', 'genre', 'name' ] }

and also notes that the value of the ref is interpreted depending on the "vehicle" in which it is being carried.

Further info

Patrice also has some great notes for this part.

Genres, cuids and a bit of AWK

2026-03-03T00:00:00Z

One of my goals this year is to learn more about the AWK programming language. I'm interested in not only the beauty of the language itself and the Unix history that surrounded it, but in how AWK shaped one of my favourite languages of all time, Perl.

In some ways AWK and CAP are similar, in that there's beauty, power and simplicity built in. These three attributes come about not least because it's about the code that you don't have to write; the language (AWK) and the framework (CAP) take care of much of the ceremony, allowing you to focus on the heart of the task at hand, and to write the smallest amount of code necessary to get the job done and minimise the maintenance burden for yourself.

Challenge

It's from the CAP context that a small challenge arose. The initial data supplied with the "bookshop" sample contains Books, Authors and Genres. But I don't like the aesthetics of the Genre IDs; they're GUIDs, as defined by the cuid aspect in the schema:

using {
  cuid,
  sap
} from '@sap/cds/common';

namespace sap.capire.bookshop;

/** Hierarchically organized Code List for Genres */
entity Genres : cuid, sap.common.CodeList {
  parent   : Association to Genres;
  children : Composition of many Genres
               on children.parent = $self;
}

The corresponding initial data (in db/data/sap.capire.bookshop.Genres.csv) looks like this (many records have been omitted for brevity):

ID,parent_ID,name
10aaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa,,Fiction
11aaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa,10aaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa,Drama
12aaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa,10aaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa,Poetry
13aaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa,10aaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa,Fantasy
131aaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa,13aaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa,Fairy Tale
132aaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa,13aaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa,Epic Fantasy
133aaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa,13aaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa,High Fantasy
134aaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa,13aaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa,Gothic
14aaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa,10aaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa,Science Fiction
141aaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa,14aaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa,Utopian and Dystopian
1411aaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa,141aaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa,Utopian
1412aaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa,141aaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa,Dystopian
14121aaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa,1412aaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa,Cyberpunk
141211aa-aaaa-aaaa-aaaa-aaaaaaaaaaaa,14121aaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa,Steampunk
...
18aaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa,10aaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa,Short Story
19aaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa,10aaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa,Graphic Novel
20aaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa,,Non-Fiction
21aaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa,20aaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa,Biography
22aaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa,21aaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa,Autobiography
23aaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa,20aaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa,Essay
24aaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa,20aaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa,Speech

This hurts my eyes, and I wanted to rebuild this dataset with integer IDs, while keeping the same hierarchy¹. As a bonus, I wanted to be able to visualise that hierarchy too.

Exploration

While I was on a similar journey to learn more about jq, I built an interactive jq REPL style data explorer, using tools in my personal development environment (i.e. my Unix shell). I described that explorer (called jqte) in the blog post A simple jq REPL with tmux, bash, vim and entr.

I found jqte so useful that I thought a similar AWK REPL style data explorer would be useful too, so I created one, called ate (AWK terminal explorer, I guess).

It's great, because:

I can see the original data (top right)
I get immediate feedback and output (left) ...
... as soon as I modify any of my AWK incantations (bottom right)

Solution

Here's the AWK script I came up with, bearing in mind that I've not written very much AWK at all, and probably most of what I have written has been very simple, in one-liners. I have written some notes below on how the script works.

#!/usr/bin/awk -f

# - Reads in a CSV file of genres
# - Emits a version with "clean(er)" IDs
# - Creates a directory tree (in /tmp/genres/) representing the hierarchy

function getparent(sid) {
  if (! parent[sid]) return ""
  return getparent(parent[sid]) "/" name[parent[sid]]
}

BEGIN {
  OFS = FS = ",";
  DIR = "genres";
  system("rm -rf /tmp/" DIR);
}

NR == 1

NR > 1 {
  sid = NR - 1
  id[$1] = sid
  name[sid] = $3
  parent[sid] = id[$2]
  print sid, parent[sid], $3
  system("mkdir -p '/tmp/" DIR getparent(sid) "/" $3 "'")
}

This script turns the above CSV records into ones that have (to me) much more pleasant identifiers:

ID,parent_ID,name
1,,Fiction
2,1,Drama
3,1,Poetry
4,1,Fantasy
5,4,Fairy Tale
6,4,Epic Fantasy
7,4,High Fantasy
8,4,Gothic
9,1,Science Fiction
10,9,Utopian and Dystopian
11,10,Utopian
12,10,Dystopian
13,12,Cyberpunk
14,13,Steampunk
...
36,1,Short Story
37,1,Graphic Novel
38,,Non-Fiction
39,38,Biography
40,39,Autobiography
41,38,Essay
42,38,Speech

You may have noticed the system( ... ) call in the script. This is related to the visualisation part of the challenge.

To address this part, I had decided not to reinvent any wheels and use the venerable tree facility available almost everywhere, which is designed to show files and directories in a hierarchical view. So all I needed was to create a hierarchy of directories, where each directory was a genre, with optional parent and children, and then display it with tree. The result looks like this:

/tmp/genres/
├── Fiction
│   ├── Adventure
│   ├── Drama
│   ├── Fantasy
│   │   ├── Epic Fantasy
│   │   ├── Fairy Tale
│   │   ├── Gothic
│   │   └── High Fantasy
│   ├── Graphic Novel
│   ├── Mystery
│   │   ├── Cozy Mystery
│   │   ├── Crime
│   │   │   ├── Detective
│   │   │   ├── Suspense
│   │   │   └── Thriller
│   │   │       ├── Legal Thriller
│   │   │       ├── Medical Thriller
│   │   │       ├── Police Procedural
│   │   │       └── Spy Thriller
│   │   └── Noir
│   │       ├── Nordic Noir
│   │       └── Tart Noir
│   ├── Poetry
│   ├── Romance
│   │   ├── Contemporary Romance
│   │   ├── Historical Romance
│   │   └── Romantic Suspense
│   ├── Science Fiction
│   │   ├── Space Opera
│   │   ├── Tech Noir
│   │   ├── Time Travel
│   │   └── Utopian and Dystopian
│   │       ├── Dystopian
│   │       │   └── Cyberpunk
│   │       │       └── Steampunk
│   │       └── Utopian
│   └── Short Story
└── Non-Fiction
    ├── Biography
    │   └── Autobiography
    ├── Essay
    └── Speech

Notes

Here are some notes on the content of the script, the small size of which is a testament to the power, beauty and simplicity of AWK and the Unix toolset, rather than any skill of mine.

Records, which contain fields, are processed one by one in AWK.
There's a PATTERN - ACTION pair approach to thinking about what you want to happen for each record.
In each pair's case, the pattern come first, then the corresponding action is defined in a { ... } block.
If the pattern matches, then the action is carried out.
There are special patterns, one of which, BEGIN, I use here to set a few things up.
There are many built-in variables; in the BEGIN section I'm setting the (input) field separator FS and the output field separator OFS; also used in this script is NR which is the number of the record being currently processed.
As well as these built-in "system" variables, the variables $1, $2, $3 and so on refer to the individual fields in a record (and $0 refers to the entire record)
AWK can make system calls, as you can also see here where I remove any previously created directory hierarchy in /tmp/genres/ at the start.
The default action is to print the entire record, which I use for the first record (pattern NR == 1) to pass through the CSV header line to the output untouched
Processing is done for each of the actual CSV records (where the pattern NR > 1 holds true, basically), where I derive a simple (integer) ID in sid and then create lookups (using AWK's array structure which is associative) for the IDs, genre names, and genre parents (where appropriate).
Variables (including arrays) benefit from autovivification ², which means I don't need to worry about declaring or initialising anything.
The print statement is where I output the CSV records each with their (and their parent's) simpler IDs.
I also invoke a mkdir command to create a directory within /tmp/genres/ to represent the current genre.
The core of this relies on the getparent function defined at the top, which recursively backtracks to derive the complete parent path.

Wrapping up

I am still fascinated by the simplicity of command line tools, flat file data processing and little languages like AWK, whose size belies its power and utility.

If you want to learn a little more about AWK, I would definitely recommend the paper Awk - A Pattern Scanning and Processing Language, concise and compact, just like AWK.

Footnotes

Yes I know there's also a corresponding .texts.csv file that I should also convert, but I would rather just delete this file, as it's just initial data for testing in all cases, anyway. I may also want to change the type of the Genres:ID element to Integer itself. But basically, this excursion was more about AWK than genre data.
It's fascinating that the linked Wikipedia page entry on autovivification specifically refers to Perl which as I've mentioned was heavily influenced by AWK as well as other contemporary languages and tools.