critcl_howto_use(n) 3.3.1 doc "C Runtime In Tcl (CriTcl)"

Simple Arguments

A function taking neither arguments nor returning results is not very useful.

1. We are now extending the command to take an argument.

2. Starting from the Basics. Edit the file "example.tcl". Remove the definition of hello. Replace it with

       critcl::cproc hello {double x} void {
   	/* double x; */
   	printf("hello world, we have %f\n", x);
       }
   

   and rebuild the package.

3. When testing the package again, entering the simple hello will fail.

   The changed command is now expecting an argument, and we gave it none.

   Retry by entering

   hello 5

   instead. Now the command behaves as expected and prints the provided value.

   Further try and enter

   hello world

   This will fail again. The command expected a real number and we gave it something decidedly not so.

   These checks (argument count, argument type) are implemented in the translation layer CriTcl generates for the C function. The function body is never invoked.

Simple Results

A function taking neither arguments nor returning results is not very useful.

1. We are now extending the command to return a result.

2. Starting from the Basics. Edit the file "example.tcl". Remove the definition of hello. Replace it with

       critcl::cproc twice {double x} double {
   	return 2*x;
       }
   

   and rebuild the package.

3. Note that the name of the command changed. Goodbye hello, hello twice.

4. Invoke

    twice 4 

   and the tclsh will print the result 8 in the terminal.

An important limitation of the commands implemented so far is that they cannot fail. The types used so far (void, double) and related scalar types can return only a value of the specified type, and nothing else. They have no ability to signal an error to the Tcl script.

We will come back to this after knowing a bit more about the more complex argument and result types.

Of interest to the eager reader: CriTcl cproc Type Reference

Range-limited Simple Arguments

1. Starting from the Basics. Edit the file "example.tcl". Remove the definition of hello. Replace it with

       critcl::cproc hello {{double > 5 < 22} x} void {
   	/* double x, range 6-21; */
   	printf("hello world, we have %f\n", x);
       }
   

   and rebuild the package.

2. When dealing with simple arguments whose range of legal values is limited to a single continuous interval extend the base type with the necessary relations (>, >=, <, and <=) and limiting values.

   Note that the limiting values have to be proper constant numbers acceptable by the base type. Symbolic values are not accepted.

   Here the argument x of the changed function will reject all values outside of the interval 6 to 21.

String Arguments

Tcl prides itself on the fact that Everything Is A String. So how are string values passed into C functions ?

1. We are now extending the command with a string argument.

2. Starting from the Basics. Edit the file "example.tcl". Remove the definition of hello. Replace it with

       critcl::cproc hello {pstring x} void {
   	/* critcl_pstring x (.s, .len, .o); */
   	printf("hello world, from %s (%d bytes)\n", x.s, x.len);
       }
   

   and rebuild the package.

3. Testing hello with any kind of argument the information is printed.

4. Of note here is that the command argument x is a structure.

5. The example uses only two of the three fields, the pointer to the string data (.s), and the length of the string (.len). In bytes, not in characters, because Tcl's internal representation of strings uses a modified UTF-8 encoding. A character consists of between 1 and TCL_UTF_MAX bytes.

6. Attention The pointers (.s) refer into data structures internal to and managed by the Tcl interpreter. Changing them is highly likely to cause subtle and difficult to track down bugs. Any and all complex arguments must be treated as Read-Only. Never modify them.

7. Use the simpler type char* if and only if the length of the string is not relevant to the command, i.e. not computed, or not used by any of the functions called from the body of the command. Its value is essentially just the .s field of pstring's structure. This then looks like

       critcl::cproc hello {char* x} void {
   	/* char* x; */
   	printf("hello world, from %s\n", x);
       }
   

String Results

Tcl prides itself on the fact that Everything Is A String. So how are string values returned from C functions ?

1. We are now giving the command a string result.

2. Starting from the Basics. Edit the file "example.tcl". Remove the definition of hello. Replace it with

       critcl::cproc twice {double x} char* {
   	char buf [lb]40[rb];
   	sprintf(buf, "%f", 2*x);
   	return buf;
       }
   

   and rebuild the package.

3. Note that the name of the command changed. Goodbye hello, hello twice.

4. Invoke

    twice 4 

   and the tclsh will print the result 8 in the terminal.

5. Attention. To the translation layer the string pointer is owned by the C code. A copy is made to become the result seen by Tcl.

   While the C code is certainly allowed to allocate the string on the heap if it so wishes, this comes with the responsibility to free the string as well. Abrogation of that responsibility will cause memory leaks.

   The type char* is recommended to be used with static string buffers, string constants and the like.

6. Conversely, to return heap-allocated strings it is recommended to use the type string instead.

   Replace the definition of twice with

   critcl::cproc twice {double x} string {
       char* buf = Tcl_Alloc (40);
       sprintf(buf, "%f", 2*x);
       return buf;
   }
   

   Now the translation layer takes ownership of the string from the C code and transfers that ownership to the Tcl interpreter. This means that the string will be released when the Tcl interpreter is done with it. The C code has no say in the lifecycle of the string any longer, and having the C code releasing the string will cause issues. Dangling pointers and associated memory corruption and crashes.

List Arguments

Even as a string-oriented language Tcl is capable of handling more complex structures. The first of it, with Tcl since the beginning are lists. Sets of values indexed by a numeric value.

In C parlance, arrays.

1. We are now extending the command with a list argument.

2. Starting from the Basics. Edit the file "example.tcl". Remove the definition of hello. Replace it with

       critcl::cproc hello {list x} void {
   	/* critcl_list x (.o, .v, .c); */
   	printf("hello world, %d elements in (%s)\n", x.c, Tcl_GetString (x.o));
       }
   

   and rebuild the package.

3. Testing hello with any kind of list argument it will print basic information about it.

4. Of note here is that the command argument x is a structure.

5. The example uses only two of the three fields, the pointer to the original Tcl_Obj* holding the list (.o), and the length of the list (.c) in elements.

   The field .v, not used above, is the C array holding the Tcl_Obj* pointers to the list elements.

6. Attention The pointers .o and .v refer into data structures internal to and managed by the Tcl interpreter. Changing them is highly likely to cause subtle and difficult to track down bugs. Any and all complex arguments must be treated as Read-Only. Never modify them.

7. As a last note, this argument type does not place any constraints on the size of the list, or on the type of the elements.

Constrained List Arguments

As mentioned at the end of section List Arguments the basic list type places no constraints on the size of the list, nor on the type of the elements.

Both kind of constraints can be done however, alone or together.

1. We are now extending the command with a length-limited list.

2. Starting from the Basics. Edit the file "example.tcl". Remove the definition of hello. Replace it with

       critcl::cproc hello {[5] x} void {
   	/* critcl_list x (.o, .v, .c); */
   	printf("hello world, %d elements in (%s)\n", x.c, Tcl_GetString (x.o));
       }
   

   and rebuild the package.

3. Testing the new command will show that only lists holding exactly 5 elements will be accepted.

4. To accept lists of any length use [] or [*]. Both forms are actually aliases of the base type, i.e. list.

5. To constrain just the type of elements, for example to type int, use

   int[]

   or

   []int

6. To combine both type and length constraints use the forms

   int[5]

   or

   [5]int

7. The last, most C-like forms of these contraints place the list indicator syntax on the argument instead of the type. I.e

   int a[]

   or

   int a[5]

Raw Tcl_Obj* Arguments

When the set of predefined argument types is not enough the oldest way of handling the situation is falling back to the structures used by Tcl to manage values, i.e. Tcl_Obj*.

1. Starting from the Basics. Edit the file "example.tcl". Remove the definition of hello. Replace it with

       critcl::cproc hello {object x} void {
   	/* Tcl_Obj* x */
   	int len;
   	char* str = Tcl_GetStringFromObj (x, &len);
   	printf("hello world, from %s (%d bytes)\n", str, len);
       }
   

   and rebuild the package.

2. Having direct access to the raw Tcl_Obj* value all functions of the public Tcl API for working with Tcl values become usable. The downside of that is that all the considerations for handling them apply as well.

   In other words, the C code becomes responsible for handling the reference counts correctly, for duplicating shared Tcl_Obj* structures before modifying them, etc.

   One thing the C code is allowed to do without restriction is to shimmer the internal representation of the value as needed, through the associated Tcl API functions. For example Tcl_GetWideIntFromObj and the like. It actually has to be allowed to do so, as the type checking done as part of such conversions is now the responsibility of the C code as well.

   For the predefined types this is all hidden in the translation layer generated by CriTcl.

   If more than one command has to perform the same kind of checking and/or conversion it is recommended to move the core of the code into proper C functions for proper sharing among the commands.

3. This is best done by defining a custom argument type using CriTcl commands. This extends the translation layer CriTcl is able to generate. The necessary conversions, type checks, etc. are then again hidden from the bulk of the application C code.

   We will come back to this.

Raw Tcl_Obj* Results

When the set of predefined result types is not enough the oldest way of handling the situation is falling back to the structures used by Tcl to manage values, i.e. Tcl_Obj*.

Two builtin types are provided for this, to handle different reference counting requirements.

1. Starting from the Basics. Edit the file "example.tcl". Remove the definition of hello. Replace it with

       critcl::cproc twice {double x} object0 {
   	return Tcl_NewDoubleObj(2*x);
       }
   

   and rebuild the package.

2. With object0 the translation layer assumes that the returned Tcl_Obj* value has a reference count of 0. I.e. a value which is unowned and unshared.

   This value is passed directly to Tcl for its use, without any changes. Tcl increments the reference count and thus takes ownership. The value is still unshared.

   It would be extremely detrimental if the translation layer had decremented the reference count before passing the value. This action would release the memory and then leave Tcl with a dangling pointer and the associated memory corruption bug to come.

3. The situation changes when the C code returns a Tcl_Obj* value with a reference count greater than 0. I.e. at least owned (by the C code), and possibly even shared. There are some object constructors and/or mutators in the public Tcl API which do that, although I do not recall their names. The example below simulates this situation by explicitly incrementing the reference count before returning the value.

4. In this case use the type object (without the trailing 0).

5. Edit the file "example.tcl" and replace the definition of twice with

       critcl::cproc twice {double x} object {
   	Tcl_Obj* result = Tcl_NewDoubleObj(2*x);
   	Tcl_IncrRefCount (result);
   	return result;
       }
   

   and rebuild the package.

6. After handing the value to Tcl, with the associated incremented reference count, the translation layer decrements the reference count, invalidating the C code's ownership and leaving the final reference count the same.

   Note, the order matters. If the value has only one reference then decrementing it before Tcl increments it would again release the value, and again leave Tcl with a dangling pointer.

   Also, not decrementing the reference count at all causes the inverse problem to the memory corruption issues of before, memory leaks.

7. Note that both types transfer ownership of the value. Their difference is just in the reference count of the value coming out of the function, and the (non-)actions having to be (not) taken to effect said transfer without causing memory issues.

Errors & Messages

1. Starting from the Basics. Edit the file "example.tcl". Remove the definition of hello. Replace it with

       critcl::cproc sqrt {
   	Tcl_Interp* interp
   	double      x
       } object0 {
   	if (x < 0) {
   	    Tcl_SetObjResult (interp, Tcl_ObjPrintf ("Expected double >=0, but got \"%d\"", x));
   	    Tcl_SetErrorCode (interp, "EXAMPLE", "BAD", "DOMAIN", NULL);
   	    return NULL;
   	}
   	return Tcl_NewDoubleObj(sqrt(x));
       }
   

   and rebuild the package.

2. In standard C-based packages commands signal errors by returning TCL_ERROR, placing the error message as the interpreter result, and maybe providing an error code via Tcl_SetErrorCode.

3. When using critcl::cproc this is limited and hidden.

4. The simple and string types for results do not allow failure. The value is returned to the translation layer, converted into the interpreter result and then reported as success (TCL_OK).

5. The object types on the other hand do allow for failure. Return a NULL value to signal failure to the translation layer, which then reports this to the interpreter via the standard TCL_ERROR.

6. Attention Setting the desired error message and code into the interpreter is still the responsibility of the function body.

Tcl_Interp* Access

1. Reread the example in the previous section.

2. Note the type Tcl_Interp* used for the first argument.

3. This type is special.

4. An argument of this type has to be the first argument of a function.

5. Using it tells CriTcl that the function needs access to the Tcl interpreter calling it. It then arranges for that to happen in the generated C code.

   Using functions from Tcl's public C API taking an interpreter argument in the function body is a situation where this is needed.

6. This special argument is not visible at the script level.

7. This special argument is not an argument of the Tcl command for the function.

8. In our example the sqrt command is called with a single argument.

9. The name of the argument can be freely chosen. It is the type which is important and triggers the special behaviour. My prefered names are ip and interp.

Binary Data Arguments

1. Starting from the Basics. Edit the file "example.tcl". Remove the definition of hello. Replace it with

       critcl::cproc hello {bytes x} void {
           /* critcl_bytes x (.s, .len, .o); */
           printf("hello world, with %d bytes \n data: ", x.len);
           for (i = 0; i < x.len; i++) {
               printf(" %02x", x.s[i]);
               if (i % 16 == 15) printf ("\ndata: ");
           }
           if (i % 16 != 0) printf ("\n");
       }
   

   and rebuild the package.

2. To deal with strings holding binary data use the type bytes. It ensures that the function sees the proper binary data, and not how Tcl is encoding it internally, as the string types would.

Constant Binary Data Results

1. Use the command critcl::cdata to create a command taking no arguments and returning a constant ByteArray value.

       # P5 3 3 255 \n ...
       critcl::cdata cross3x3pgm {
   	80 52 32 51 32 51 32 50 53 53 10
   	0   255 0
   	255 255 255
   	0   255 0
       }
   

Tcl Runtime Version

1. See and reread the basic package for the introduction of the commands referenced below.

2. Use the command critcl::tcl to tell CriTcl the minimal version of Tcl the package is to be used with.

   This determines which Tcl headers all files are compiled against, and what version of the public Tcl API is available to the C code.

   Currently 8.4, 8.5 and 8.6 are supported.

   If not specified 8.4 is assumed.

Additional Tcl Code

1. Starting from the Basics. Edit the file "example.tcl". Remove the definition of hello. Replace it with

       critcl::cproc greetings::hello {} void {
   	printf("hello world\n");
       }
       critcl::cproc greetings::hi {} void {
   	printf("hi you\n");
       }
   

   and rebuild the package.

2. The command hello is now available as greetings::hello, and a second command greetings::hi was added.

3. Tcl has automatically created the namespace greetings.

4. Create a file "example-policy.tcl" and enter

       namespace eval greetings {
   	namespace export hello hi
   	namespace ensemble create
       }
   

   into it.

5. Edit "example.tcl". Add the code

       critcl::tsources example-policy.tcl
   

   and rebuild the package.

6. The added Tcl code makes greetings available as an ensemble command.

   The commands in the namespace have been registered as methods of the ensemble.

   They can now be invoked as

       greetings hello
       greetings hi
   

7. The Tcl builtin command string is an ensemble as well, as is clock.

New commands: critcl::tsources

Debugging Support

1. See and reread the basic package for the introduction of the commands referenced below.

2. Use the command critcl::debug to activate various features supporting debugging.

       critcl::debug memory  ;# Activate Tcl memory debugging (-DTCL_MEM_DEBUG)
       critcl::debug symbols ;# Activate building and linking with debugger symbols (-g)
       critcl::debug all     ;# Shorthand for both `memory` and `symbols`.
   

Install The Package

1. Starting from the Basics.

2. Use an interactive tclsh seesion to determine the value of info library.

   For the purpose of this HowTo assume that this path is "/home/aku/opt/ActiveTcl/lib/tcl8.6"

3. Invoke the critcl application in a terminal, using

       critcl -libdir /home/aku/opt/ActiveTcl/lib/tcl8.6 -pkg example.tcl
   

4. The package is now build and installed into the chosen directory.

      % find /home/aku/opt/ActiveTcl/lib/tcl8.6/example/
       /home/aku/opt/ActiveTcl/lib/tcl8.6/example/
       /home/aku/opt/ActiveTcl/lib/tcl8.6/example/pkgIndex.tcl
       /home/aku/opt/ActiveTcl/lib/tcl8.6/example/critcl-rt.tcl
       /home/aku/opt/ActiveTcl/lib/tcl8.6/example/license.terms
       /home/aku/opt/ActiveTcl/lib/tcl8.6/example/linux-x86_64
       /home/aku/opt/ActiveTcl/lib/tcl8.6/example/linux-x86_64/example.so
       /home/aku/opt/ActiveTcl/lib/tcl8.6/example/teapot.txt
   

Default Values For Arguments

1. Reread the example of the main section. Note specifically the line

       int {level ZSTD_CLEVEL_DEFAULT}
   

2. This line demonstrates that critcl::cproc arguments allowed to have default values, in the same vein as proc arguments, and using the same syntax.

3. Attention Default values have to be legal C rvalues and match the C type of the argument.

   They are literally pasted into the generated C code.

   They bypass any argument validation done in the generated translation layer. This means that it is possible to use a value an invoker of the command cannot use from Tcl.

4. This kind of in-band signaling of a default versus a regular argument is however not necessary.

   Look at

       critcl::cproc default_or_not {int {x 0}} void {
   	if !has_x {
   	    printf("called with default\n");
   	    return
   	}
   	printf("called with %d\n", x);
       }
   

   Any argument x with a default causes CriTcl to create a hidden argument has_x, of type int (boolean). This argument is set to 1 when x was filled from defaults, and 0 else.

Custom Argument Validation

1. Starting from the base wrapper. Edit the file "example.tcl". Replace the entire compress function with

       critcl::argtype zstd_compression_level {
           /* argtype: `int` */
           if (Tcl_GetIntFromObj (interp, @@, &@A) != TCL_OK) return TCL_ERROR;
           /* additional validation */    
           int max = ZSTD_maxCLevel();
           if ((@A < 1) || (@A > max)) {
               Tcl_SetObjResult (interp,
                   Tcl_ObjPrintf ("zstd compression level must be integer between 1 and %d", max));
               return TCL_ERROR;
           }
           /* @@: current objv[] element
           ** @A: name of argument variable for transfer to C function
           ** interp: predefined variable, access to current interp - error messages, etc.
           */
       } int int ;# C types of transfer variable and function argument.
       critcl::cproc compress {
           Tcl_Interp*            ip
           bytes                  data
           zstd_compression_level {level ZSTD_CLEVEL_DEFAULT}
       } object0 {
           /* critcl_bytes data; (.s, .len, .o) */
           /* int level; validated to be in range 1...ZSTD_maxCLevel() */
           Tcl_Obj* error_message;
           size_t dest_sz  = ZSTD_compressBound (data.len);
           void*  dest_buf = Tcl_Alloc(dest_sz);
           if (!dest_buf) {
               error_message = Tcl_NewStringObj ("can't allocate memory to compress data", -1);
               goto err;
           }
           size_t compressed_size = ZSTD_compress (dest_buf, dest_sz,
                                                   data.s,   data.len,
                                                   level);
           if (ZSTD_isError (compressed_size)) {
               Tcl_Free(dest_buf);
               error_message = Tcl_ObjPrintf ("zstd encoding error: %s",
                                              ZSTD_getErrorName (compressed_size));
               goto err;
           }
           Tcl_Obj* compressed = Tcl_NewByteArrayObj (dest_buf, compressed_size);
           Tcl_Free (dest_buf);
           return compressed;
       err:
           Tcl_SetObjResult (ip, error_message);
           return 0;
       }
   

   and rebuild the package.

   In the original example the level argument of the function was validated in the function itself. This may detract from the funtionality of interest itself, especially if there are lots of arguments requiring validation. If the same kind of argument is used in multiple places this causes code duplication in the functions as well.

   Use a custom argument type as defined by the modification to move this kind of validation out of the function, and enhance readability.

   Code duplication however is only partially adressed. While there is no duplication in the visible definitions the C code of the new argument type is replicated for each use of the type.

2. Now replace the argtype definition with

       critcl::code {
           int GetCompressionLevel (Tcl_Interp* interp, Tcl_Obj* obj, int* level)
           {
               if (Tcl_GetIntFromObj (interp, obj, level) != TCL_OK) return TCL_ERROR;
               int max = ZSTD_maxCLevel();
               if ((*level < 1) || (*level > max)) {
                   Tcl_SetObjResult (interp,
                       Tcl_ObjPrintf ("zstd compression level must be integer between 1 and %d", max));
                   return TCL_ERROR;
               }
               return TCL_OK;
           }
       }
       critcl::argtype zstd_compression_level {
           if (GetCompressionLevel (@@, &@A) != TCL_OK) return TCL_ERROR;
       } int int
   

   and rebuild the package.

   Now only the calls to the new validation function are replicated. The function itself exists only once.

Separating Local C Sources

1. Starting from the end of the previous section. Edit the file "example.tcl".

2. Save the contents of the critcl::ccode block into a file "example.c" and then replace the entire block with

       critcl::csources example.c
       critcl::ccode {
   	extern int GetCompressionLevel (Tcl_Interp* interp, Tcl_Obj* obj, int* level);
       }
   

   When mixing C and Tcl code the different kind of indentation rules for these languages may come into strong conflict. Further, very large blocks of C code may reduce overall readability.

3. The examples fixes this by moving the code block into a local C file and then registering this file with CriTcl. When building the package CriTcl arranges to build all such registered C files as well.

4. Attention. The C code is now in a separate compilation unit. The example declares the exported function so that the cprocs are again able to see and use it.

5. Now go a step further. Save the declaration into a file "example.h", and then use

       critcl::include example.h
   

   to import it. Note that this is just a shorthand for

    critcl::ccode {
   	#include "example.h"
       }
   

6. As an alternative solution, start from the beginning of the section and move the entire original critcl::ccode block into a file "example-check.tcl".

   Then replace it with

       critcl::source example-check.tcl
   

   to import it into the main code again.

   Attention Tcl's builtin command source is not suitable for importing the separate file due to how CriTcl uses the information from info script to key various internal datastructures.

Very Simple Results

1. Starting from the end of the validation section. Edit the file "example.tcl". Add the code below, just before the compress command.

       critcl::cconst version   char* ZSTD_VERSION_STRING
       critcl::cconst min-level int   1
       critcl::cconst max-level int   ZSTD_maxCLevel()
   

   and rebuild the package.

2. These declarations create three additional commands, each returning the specified value. A fixed string, an integer, and a function call returning an integer.

3. Attention The values have to be legal C rvalues and match the C type of the result. They are literally pasted into the generated C code.

4. When using critcl::cconst CriTcl is aware that the result of the function does not depend on any parameters and is computed in a single C expression.

   This enables it do to away with the internal helper function it would need and generate if critcl::cproc had been used instead. For example

       critcl::cproc version {} char* {
   	return ZSTD_VERSION_STRING;
       }
   

Structure Arguments

1. For all that this is a part of how to Use External Libraries, for the demonstratation only the basics are needed.

2. Starting from the Basics. Edit the file "example.tcl". Remove the definition of hello. Replace it with

       critcl::ccode {
   	typedef struct vec2 {
   	    double x;
   	    double y;
   	} vec2;
   	typedef vec2* vec2ptr;
   	int
   	GetVecFromObj (Tcl_Interp* interp, Tcl_Obj* obj, vec2ptr* vec)
   	{
   	    int len;
   	    if (Tcl_ListObjLength (interp, obj, &len) != TCL_OK) return TCL_ERROR;
   	    if (len != 2) {
   		Tcl_SetObjResult (interp, Tcl_ObjPrintf ("Expected 2 elements, got %d", len));
   		return TCL_ERROR;
   	    }
   	    Tcl_Obj* lv[2];
   	    if (Tcl_ListObjGetElements (interp, obj, &lv) != TCL_OK) return TCL_ERROR;
   	    double x, y;
   	    if (Tcl_GetDoubleFromObj (interp, lv[0], &x) != TCL_OK) return TCL_ERROR;
   	    if (Tcl_GetDoubleFromObj (interp, lv[1], &y) != TCL_OK) return TCL_ERROR;
   	    *vec = Tcl_Alloc (sizeof (vec2));
   	    (*vec)->x = x;
   	    (*vec)->y = y;
   	    return TCL_OK;
   	}
       }
       critcl::argtype vec2 {
   	if (GetVecFromObj (interp, @@, &@A) != TCL_OK) return TCL_ERROR;
       } vec2ptr vec2ptr
       critcl::argtyperelease vec2 {
   	/* @A : C variable holding the data to release */
   	Tcl_Free ((char*) @A);
       }
       critcl::cproc norm {vec2 vector} double {
   	double norm = hypot (vector->x, vector->y);
   	return norm;
       }
   

   and rebuild the package.

3. The structure to pass as argument is a 2-dimensional vector. It is actually passed in as a pointer to a vec2 structure. This pointer is created by the GetVecFromObj function. It allocates and fills the structure from the Tcl value, which has to be a list of 2 doubles. The bulk of the code in GetVecFromObj is for verifying this and extracting the doubles.

4. The argtyperelease code releases the pointer when the C function returns. In other words, the pointer to the structure is owned by the translation layer and exists only while the function is active.

5. While working this code has two disadvantages. First there is memory churn. Each call of norm causes the creation and release of a temporary vec2 structure for the argument. Second is the need to always extract the data from the Tcl_Obj* value.

   Both can be done better.

   We will come back to this after explaining how to return structures to Tcl.

Structure Results

1. Starting from the end of the previous section.

2. Edit the file "example.tcl" and add the following code, just after the definition of the norm command.

       critcl::resulttype vec2 {
   	/* rv: result value of function, interp: current Tcl interpreter */
   	if (rv == NULL) return TCL_ERROR;
   	Tcl_Obj* lv[2];
   	lv[0] = Tcl_NewDoubleObj (rv->x);
   	lv[1] = Tcl_NewDoubleObj (rv->y);
   	Tcl_SetObjResult (interp, Tcl_NewListObj (2, lv));
   	Tcl_Free (rv);
   	return TCL_OK;
       } vec2ptr ;# C result type
       critcl::cproc add {vec2 a vec2 b} vec2 {
   	vec2ptr z = Tcl_Alloc (sizeof (vec2));
   	z->x = a->x + b->x;
   	z->y = a->y + b->y;
   	return z;
       }
   

   and rebuild the package.

3. The new command add takes two vectors and return the element-wise sum of both as a new vector.

4. The function allocates and initializes a structure and hands it over to the translation layer. Which in turn constructs a Tcl list of 2 doubles from it, sets that as the command's result and at last discards the allocated structure again.

5. While working this code has two disadvantages. First there is memory churn. Each call of add causes the creation and release of three temporary vec2 structures. One per argument, and one for the result. Second is the need to always construct a complex Tcl_Obj* value from the structure.

   Both can be done better. This is explained in the next section.

Structure Types

1. Starting from the end of the previous section.

2. Edit the file "example.tcl".

3. Remove the entire functionality (type definitions, related C code, and cprocs). Replace it with

       critcl::ccode {
   	typedef struct vec2 {
   	    double x;
   	    double y;
   	} vec2;
   	typedef vec2* vec2ptr;
   	/* -- Core vector structure management -- */
   	static vec2ptr Vec2New (double x, double y) {
   	    vec2ptr vec = Tcl_Alloc (sizeof (vec2));
   	    vec->x = x;
   	    vec->y = y;
   	    return vec;
   	}
   	static vec2ptr Vec2Copy (vec2ptr src) {
   	    vec2ptr vec = Tcl_Alloc (sizeof (vec2));
   	    *vec = *src
   	    return vec;
   	}
   	static void Vec2Release (vec2ptr vec) {
   	    Tcl_Free ((char*) vec);
   	}
   	/* -- Tcl value type for vec2 -- Tcl_ObjType -- */
   	static void Vec2Free     (Tcl_Obj* obj);
   	static void Vec2StringOf (Tcl_Obj* obj);
   	static void Vec2Dup      (Tcl_Obj* obj, Tcl_Obj* dst);
   	static int  Vec2FromAny  (Tcl_Interp* interp, Tcl_Obj* obj);
   	Tcl_ObjType vec2_objtype = {
   	    "vec2",
   	    Vec2Free,
   	    Vec2Dup,
   	    Vec2StringOf,
   	    Vec2FromAny
   	};
   	static void Vec2Free (Tcl_Obj* obj) {
   	    Vec2Release ((vec2ptr) obj->internalRep.otherValuePtr);
   	}
   	static void Vec2Dup (Tcl_Obj* obj, Tcl_Obj* dst) {
   	    vec2ptr vec = (vec2ptr) obj->internalRep.otherValuePtr;
   	    dst->internalRep.otherValuePtr = Vec2Copy (vec);
   	    dst->typePtr                   = &vec2_objtype;
   	}
   	static void Vec2StringOf (Tcl_Obj* obj) {
   	    vec2ptr vec = (vec2ptr) obj->internalRep.otherValuePtr;
   	    /* Serialize vector data to string (list of two doubles) */
   	    Tcl_DString      ds;
   	    Tcl_DStringInit (&ds);
   	    char buf [TCL_DOUBLE_SPACE];
   	    Tcl_PrintDouble (0, vec->x, buf); Tcl_DStringAppendElement (&ds, buf);
   	    Tcl_PrintDouble (0, vec->y, buf); Tcl_DStringAppendElement (&ds, buf);
   	    int length = Tcl_DStringLength (ds);
   	    /* Set string representation */
   	    obj->length = length;
   	    obj->bytes  = Tcl_Alloc(length+1);
   	    memcpy (obj->bytes, Tcl_DStringValue (ds), length);
   	    obj->bytes[length] = '\0';
   	    /*
   	    ** : package require critcl::cutil ;# get C utilities
   	    ** : critcl::cutil::alloc          ;# Activate allocation utilities
   	    ** : (Internally cheaders, include)
   	    ** : Then all of the above can be written as STREP_DS (obj, ds);
   	    ** : STREP_DS = STRing REP from DString
   	    */
   	    Tcl_DStringFree (&ds);
   	}
   	static int Vec2FromAny (Tcl_Interp* interp, Tcl_Obj* obj) {
   	    /* Change intrep of obj to vec2 structure.
   	    ** A Tcl list of 2 doubles is used as an intermediary intrep.
   	    */
   	    int len;
   	    if (Tcl_ListObjLength (interp, obj, &len) != TCL_OK) return TCL_ERROR;
   	    if (len != 2) {
   		Tcl_SetObjResult (interp, Tcl_ObjPrintf ("Expected 2 elements, got %d", len));
   		return TCL_ERROR;
   	    }
   	    Tcl_Obj* lv[2];
   	    if (Tcl_ListObjGetElements (interp, obj, &lv) != TCL_OK) return TCL_ERROR;
   	    double x, y;
   	    if (Tcl_GetDoubleFromObj (interp, lv[0], &x) != TCL_OK) return TCL_ERROR;
   	    if (Tcl_GetDoubleFromObj (interp, lv[1], &y) != TCL_OK) return TCL_ERROR;
   	    obj->internalRep.otherValuePtr = (void*) Vec2New (x, y);
   	    obj->typePtr                   = &vec2_objtype;
   	    return TCL_OK;
   	}
   	/* -- (un)packing structures from/into Tcl values -- */
   	int GetVecFromObj (Tcl_Interp* interp, Tcl_Obj* obj, vec2ptr* vec)
   	{
   	    if (obj->typePtr != &vec2_objtype) {
   		if (Vec2FromAny (interp, obj) != TCL_OK) return TCL_ERROR;
   	    }
   	    *vec = (vec2ptr) obj->internalRep.otherValuePtr;
   	    return TCL_OK;
   	}
   	Tcl_Obj* NewVecObj (vec2ptr vec) {
   	    Tcl_Obj* obj = Tcl_NewObj ();
   	    Tcl_InvalidateStringRep (obj);
   	    obj->internalRep.otherValuePtr = Vec2Copy (vec);
   	    obj->typePtr                   = &vec2_objtype;
   	    return obj;
   	}
       }
       critcl::argtype vec2 {
   	if (GetVecFromObj (interp, @@, &@A) != TCL_OK) return TCL_ERROR;
       } vec2ptr vec2ptr
       critcl::resulttype vec2 {
   	/* rv: result value of function, interp: current Tcl interpreter */
   	Tcl_SetObjResult (interp, NewVecObj (&rv));
   	return TCL_OK;
       } vec2
       critcl::cproc norm {vec2 vector} double {
   	double norm = hypot (vector->x, vector->y);
   	return norm;
       }
       critcl::cproc add {vec2 a vec2 b} vec2 {
   	vec2 z;
   	z.x = a->x + b->x;
   	z.y = a->y + b->y;
   	return z;
       }
   

   and rebuild the package.

4. This implements a new Tcl_ObjType to handle vec2 structures. With it vec2 structures are become usable as internal representation (intrep of Tcl_Obj* values.

   The two functions NewVecObj and GetVecFromObj pack and unpack the structures from and into Tcl_Obj* values. The latter performs the complex deserialization into a structure if and only if needed, i.e. when the TclObj* value has no intrep, or the intrep for a different type. This process of changing the intrep of a Tcl value is called shimmering.

   Intreps cache the interpretation of Tcl_Obj* values as a specific kind of type. Here vec2. This reduces conversion effort and memory churn, as intreps are kept by the Tcl interpreter as long as possible and needed.

5. The arguments of norm and add are now converted once on entry, if not yet in the proper type, or not at all, if so.

6. Attention. This example has the issue of passing result structures by value through the stack, and then packing a copy into a Tcl_Obj* value. While this is no trouble for structures as small as vec2 larger structures may pose a problem.

   We will address this in the next section.

Packages: critcl::cutil

Large Structures

1. Starting from the end of the previous section.

2. Edit the file "example.tcl".

3. Describing each individual change is too complex. The following is easier.

4. Save the file, then replace the entire functionality with the following.

5. After that use a diff of your choice to compare the files and see the critical changes.

       critcl::ccode {
   	typedef struct vec2 {
   	    unsigned int rc;
   	    double x;
   	    double y;
   	} vec2;
   	typedef vec2* vec2ptr;
   	/* -- Core vector structure management -- */
   	static vec2ptr Vec2New (double x, double y) {
   	    vec2ptr vec = Tcl_Alloc (sizeof (vec2));
   	    vec->rc = 0;
   	    vec->x = x;
   	    vec->y = y;
   	    return vec;
   	}
   	static vec2ptr Vec2Copy (vec2ptr src) {
   	    scr->rc ++;
   	    return src;
   	}
   	static void Vec2Release (vec2ptr vec) {
   	    if (vec->rc > 1) {
   		vec->rc --;
   		return;
   	    }
   	    
   	    Tcl_Free ((char*) vec);
   	}
   	/* -- Vector obj type -- */
   	static void Vec2Free     (Tcl_Obj* obj);
   	static void Vec2StringOf (Tcl_Obj* obj);
   	static void Vec2Dup      (Tcl_Obj* obj, Tcl_Obj* dst);
   	static int  Vec2FromAny  (Tcl_Interp* interp, Tcl_Obj* obj);
   	Tcl_ObjType vec2_objtype = {
   	    "vec2",
   	    Vec2Free,
   	    Vec2Dup,
   	    Vec2StringOf,
   	    Vec2FromAny
   	};
   	static void Vec2Free (Tcl_Obj* obj) {
   	    Vec2Release ((vec2ptr) obj->internalRep.otherValuePtr);
   	}
   	static void Vec2Dup (Tcl_Obj* obj, Tcl_Obj* dst) {
   	    vec2ptr vec = (vec2ptr) obj->internalRep.otherValuePtr;
   	    dst->internalRep.otherValuePtr = Vec2Copy (vec);
   	    dst->typePtr                   = &vec2_objtype;
   	}
   	static void Vec2StringOf (Tcl_Obj* obj) {
   	    vec2ptr vec = (vec2ptr) obj->internalRep.otherValuePtr;
   	    /* Serialize vector data to string (list of two doubles) */
   	    Tcl_DString      ds;
   	    Tcl_DStringInit (&ds);
   	    char buf [TCL_DOUBLE_SPACE];
   	    Tcl_PrintDouble (0, vec->x, buf); Tcl_DStringAppendElement (&ds, buf);
   	    Tcl_PrintDouble (0, vec->y, buf); Tcl_DStringAppendElement (&ds, buf);
   	    int length = Tcl_DStringLength (ds);
   	    /* Set string representation */
   	    obj->length = length;
   	    obj->bytes  = Tcl_Alloc(length+1);
   	    memcpy (obj->bytes, Tcl_DStringValue (ds), length);
   	    obj->bytes[length] = '\0';
   	    /*
   	    ** : package require critcl::cutil ;# get C utilities
   	    ** : critcl::cutil::alloc          ;# Activate allocation utilities
   	    ** : (Internally cheaders, include)
   	    ** : Then all of the above can be written as STREP_DS (obj, ds);
   	    ** : STREP_DS = STRing REP from DString
   	    */
   	    Tcl_DStringFree (&ds);
   	}
   	static int Vec2FromAny (Tcl_Interp* interp, Tcl_Obj* obj) {
   	    /* Change internal rep of obj to vector structure.
   	    ** A Tcl list of 2 doubles is used as intermediary int rep.
   	    */
   	    int len;
   	    if (Tcl_ListObjLength (interp, obj, &len) != TCL_OK) return TCL_ERROR;
   	    if (len != 2) {
   		Tcl_SetObjResult (interp, Tcl_ObjPrintf ("Expected 2 elements, got %d", len));
   		return TCL_ERROR;
   	    }
   	    Tcl_Obj* lv[2];
   	    if (Tcl_ListObjGetElements (interp, obj, &lv) != TCL_OK) return TCL_ERROR;
   	    double x, y;
   	    if (Tcl_GetDoubleFromObj (interp, lv[0], &x) != TCL_OK) return TCL_ERROR;
   	    if (Tcl_GetDoubleFromObj (interp, lv[1], &y) != TCL_OK) return TCL_ERROR;
   	    obj->internalRep.otherValuePtr = (void*) Vec2New (x, y);
   	    obj->typePtr                   = &vec2_objtype;
   	    return TCL_OK;
   	}
   	/* (un)packing structures from/into Tcl values -- */
   	int GetVecFromObj (Tcl_Interp* interp, Tcl_Obj* obj, vec2ptr* vec)
   	{
   	    if (obj->typePtr != &vec2_objtype) {
   		if (VecFromAny (interp, obj) != TCL_OK) return TCL_ERROR;
   	    }
   	    *vec = (vec2ptr) obj->internalRep.otherValuePtr;
   	    return TCL_OK;
   	}
   	Tcl_Obj* NewVecObj (vec2ptr vec) {
   	    Tcl_Obj* obj = Tcl_NewObj ();
   	    Tcl_InvalidateStringRep (obj);
   	    obj->internalRep.otherValuePtr = Vec2Copy (vec);
   	    obj->typePtr                   = &vec2_objtype;
   	    return obj;
   	}
       }
       critcl::argtype vec2 {
   	if (GetVecFromObj (interp, @@, &@A) != TCL_OK) return TCL_ERROR;
       } vec2ptr vec2ptr
       critcl::resulttype vec2 {
   	/* rv: result value of function, interp: current Tcl interpreter */
   	Tcl_SetObjResult (interp, NewVecObj (rv));
   	return TCL_OK;
       } vec2ptr
       critcl::cproc norm {vec2 vector} double {
   	double norm = hypot (vector->x, vector->y);
   	return norm;
       }
       critcl::cproc add {vec2 a vec2 b} vec2 {
   	return Vec2New (a->x + b->x, a->y + b->y);
       }
   

6. The vec2 structure is now reference counted.

7. The core management functions, i.e. Vec2New, Vec2Copy, and Vec2Release are changed to maintain that reference count. Starting at 0 on creation, copies increment, and releases decrement. A structure is actually only freed when its reference count falls to 0 or below.

8. vec2 results are changed to pointers, easily passed back through the stack. The modified translation layer just wraps it into a Tcl_Obj* value.

9. Attention. Duplicating such a Tcl_Obj* does not duplicate the referenced vec2 structure anymore, just adds a reference.

10. Regarding diff commands, I know of two graphical diffs for Tcl/Tk, TkDiff, and Eskil.

Packages: critcl::cutil

External Structures

1. Handle structures provided by external libraries using either Structure Types or Large Structures as template.

2. Attention. The choice is with the developer.

   This is true even if the external structure is not reference counted by itself.

   To reference count a structure S without such simply wrap S into a local structure which provides the reference count and has a field for S (pointer or value).

3. Attention Opaque external types, i.e. pointers to structures with hidden fields, can also be handled by the given templates.

External Enumerations

This section demonstrates how to convert from any kind of enumeration provided by an external library to Tcl strings, and the converse.

1. For all that this is a part of how to Use External Libraries, for the demonstratation only the basics are needed.

2. Starting from the Basics. Edit the file "example.tcl". Remove the definition of hello. Replace it with

       package require critcl::emap
       # no header included due to use of literal ints instead of symbolic names
       critcl::emap::def yaml_sequence_style_t {
   	any   0
   	block 1
   	flow  2
       }
       # encode: style to int
       critcl::cproc encode {yaml_sequence_style_t style} int {
   	return style;
       }
       # decode: int to style
       critcl::cproc decode {int style} yaml_sequence_style_t {
   	return style;
       }
   

   and rebuild the package.

3. The map converts between the Tcl level strings listed on the left side to the C values on the right side, and the reverse.

4. It automatically generates critcl::argtype and critcl::resulttype definitions.

5. Attention Like the default values for cproc arguments, and the results for cconst definitions the values on the right side have to be proper C rvalues. They have to match C type int.

   In other words, it is perfectly ok to use the symbolic names provided by the header file of the external library.

   Attention This however comes at a loss in efficiency. As CriTcl then has no insight into the covered range of ints, gaps, etc. it has to perform a linear search when mapping from C to Tcl. When it knows the exact integer values it can use a table lookup instead.

   Attention It also falls back to a search if a lookup table would contain more than 50 entries.

Packages: critcl::emap

External Bitsets/Bitmaps/Flags

This section demonstrates how to convert from any kind of bit-mapped flags provided by an external library to lists of Tcl strings, and the converse.

1. For all that this is a part of how to Use External Libraries, for the demonstratation only the basics are needed.

2. Starting from the Basics. Edit the file "example.tcl". Remove the definition of hello. Replace it with

       # http://man7.org/linux/man-pages/man7/inotify.7.html
       
       package require critcl::bitmap
       # critcl::cheaders - n/a, header is in system directories
       critcl::include sys/inotify.h
       
       critcl::bitmap::def tcl_inotify_events {
   	accessed       IN_ACCESS
   	all            IN_ALL_EVENTS
   	attribute      IN_ATTRIB
   	closed         IN_CLOSE
   	closed-nowrite IN_CLOSE_NOWRITE
   	closed-write   IN_CLOSE_WRITE
   	created        IN_CREATE
   	deleted        IN_DELETE
   	deleted-self   IN_DELETE_SELF
   	dir-only       IN_ONLYDIR
   	dont-follow    IN_DONT_FOLLOW
   	modified       IN_MODIFY
   	move           IN_MOVE
   	moved-from     IN_MOVED_FROM
   	moved-self     IN_MOVE_SELF
   	moved-to       IN_MOVED_TO
   	oneshot        IN_ONESHOT
   	open           IN_OPEN
   	overflow       IN_Q_OVERFLOW
   	unmount        IN_UNMOUNT
       } {
   	all closed move oneshot
       }
       
       # encode: flag set to int
       critcl::cproc encode {tcl_inotify_events e} int {
   	return e;
       }
       # decode: int to flag set
       critcl::cproc decode {int e} tcl_inotify_events {
   	return e;
       }
   

   and rebuild the package.

3. The map converts between lists of the Tcl level strings listed on the left side to the bit-union of the C values on the right side, and the reverse.

   It is noted that the four strings all, closed, move, and oneshot cannot be converted from C flags to list of strings, only from list to bits.

4. It automatically generates critcl::argtype and critcl::resulttype definitions.

5. Attention Like the default values for cproc arguments, and the results for cconst definitions the values on the right side have to be proper C rvalues. They have to match C type int.

   In other words, it is perfectly ok to use the symbolic names provided by the header file of the external library. As shown.

Packages: critcl::bitmap

Non-standard header/library locations

1. See and reread the basic wrapper package for the introduction of the commands referenced below.

2. Attention Relative paths will be resolved relative to the location of the ".tcl" file containing the CriTcl commands.

3. Use the command critcl::cheaders to tell CriTcl about non-standard locations for header files.

   Multiple arguments are allowed, and multiple calls as well. The information accumulates.

   Arguments of the form "-Idirectory" register the directory directly.

   For arguments of the form "path" the directory holding the path is registered. In other words, it is assumed to be the full path of a header file, and not a directory.

       critcl::cheaders -I/usr/local/include
       critcl::cheaders local/types.h
       critcl::cheaders other-support/*.h
   

4. Use the command critcl::include to actually use a specific header file.

5. Use the command critcl::clibraries to tell CriTcl about non-standard locations for shared libaries, and about shared libaries to link to

   Multiple arguments are allowed, and multiple calls as well. The information accumulates.

   Arguments of the form "-Ldirectory" register a directory.

   Arguments of the form "-lname" register a shared libary to link to by name. The library will be looked for in both standard and registered directories.

   Arguments of the form "-path" register a shared libary to link to by full path.

       critcl::clibraries -L/usr/lib/x86_64-linux-gnu
       critcl::clibraries -lzstd
       critcl::clibraries /usr/lib/x86_64-linux-gnu/libzstd.so
   

6. On Mac OS X use the command critcl::framework to name the frameworks to use in the package.

   Attention Using the command on other platforms is ok, and will be ignored.

7. Not answered in the above is how to find the necessary paths if they are not fixed across machines or platforms.

   We will come back to this.

Non-standard compile/link configuration

1. See and reread the basic wrapper package for the introduction of the commands referenced below.

2. Use the command critcl::cflags to provide additional, non-standard flags to the compiler.

       critcl::cflags -DBYTE_ORDER=bigendian
   

3. Use the command critcl::ldflags to provide additional, non-standard flags to the linker.

       critcl::ldflags -
   

4. Not answered in the above is how to determine such flags if they are not fixed across machines or platforms.

   This is addressed by the next section.

Querying the compilation environment

1. Use the command critcl::check to immediately check if a piece of C code can compiled successfully as a means of querying the compiler configuration itself.

       if {[critcl::check {
           #include <FOO.h>
       }]} {
           Do stuff with FOO.h present.
       } else {
           Do stuff without FOO.h
       }
   

   All header and library paths which were registered with CriTcl before using critcl::check take part in the attempted compilation.

   Use the package critcl::util and various convenience commands it provides.

2. Use the full Power of Tcl (tm) itself.

Shared C Code

1. See and reread the basic wrapper package for the introduction of the commands referenced below.

2. Use the command critcl::ccode to write C code residing outside of cproc bodies.

3. Or, alternatively, place the C code into one or more ".c" files and use the command critcl::csources to register them with CriTcl for compilation.

4. This topic is also treated in section Separating Local C Sources.

Author, License, Description, Keywords

1. See and reread the basic package for the introduction of the commands referenced below.

2. Use the command critcl::license to set the package license.

   Use the same command to set the package author.

   Both arguments are free form text.

3. Use the command critcl::summary to set a short package description.

4. Use the command critcl::description to set a longer package description.

   The arguments of both commands are free form text.

5. Use the command critcl::subject to set one or more keywords.

   Attention Contrary to the other commands the arguments accumulate.

6. All the commands are optional.

7. Their information is not placed into the generated C code.

8. In package mode the information is placed into the file "teapot.txt" of the generated package.

9. This file serves as integration point for Teapot, the package system of ActiveTcl.

Get Critcl Application Help

1. Invoke the command

       critcl -help
   

   in a terminal to get help about the critcl application.

Supported Targets & Configurations

1. Invoke the application as

       critcl -show
   

   in a terminal to get the detailed configuration of the target for the current platform.

2. Invoke the application as

       critcl -show -target NAME
   

   in a terminal to get the detailed configuration of the named target.

3. Invoke the application as

       critcl -targets
   

   in a terminal to get a list of the available targets.

Building A Package

1. Start at section Basics.

Building A Package For Debugging

1. Start at section Basics.