pynapl

KO | EN

gitlite — search

#typescript #ai-agents #ai #dsh-plugin #deepseek-harness #open-source #cli #claude-code #codex #developer-tools #react #windows

★ 86 Open GitHub ↗

Dyalog APL ←→ Python interface

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Py'n'APL: APL-Python interface

This is an interface between Dyalog APL and Python. It allows Python
code to be accessed from APL, and vice versa.

#### Requirements:

- Dyalog APL version 16.0 Unicode
 - Python 2.7.9 or higher, or Python 3.4 or higher.

## User manual

### Accessing Python from APL

The APL side of the interface is located in `Py.dyalog`. 
It can be loaded into the workspace using:

```apl
2⎕FIX'/path/to/pynapl/Py.dyalog'
```

Or using Link if .NET is available

```apl
]Link.Create Py /path/to/pynapl/Py.dyalog
```

Note that it expects the included Python scripts to be
in the same directory as the `Py.dyalog` file.

#### Starting a Python interpreter

To start a Python interpreter, make a new instance of the
`Py.Py` class. This will start a Python instance in the background,
and connect to it. On Unix, this is done using two pipes; on Windows
this is done using a TCP connection.

```apl
py ← ⎕NEW Py.Py
```

The resulting object can be used to interact with the Python
interpreter. Once the object is destroyed, the Python interpreter
associated with it will also be shut down.

There are several different options that can be given to the Py
class, namely:

| Option | Argument | Purpose |
| --- | --- | --- |
| `Attach` | ignored | Do not start up a Python instance, but allow attachment to one that is already running. An input and output pipe will be given (or a port number in TCP mode), and it will wait for a connection from the Python side. The Python side can be told to connect using `APL.client(in,out)` (or `APL.tcpclient(port)`). |
| `ForceTCP` | boolean | Use TCP mode even on Unix. This may be necessary for non-standard interpreters. |
| `PyPath` | path to an interpreter | Start the Python interpreter given in the argument, instead of the system one. |
| `ArgFmt` | string, where `⍎` will be replaced by the path to the slave script, `→` by the input pipe file (or `TCP` if in TCP mode), and `←` by the output pipe file (or port number if in TCP mode) | When used in combination with `PyPath`, use a custom argument format rather than the standard one. |
| `Version` | major Python version (2 or 3) | Start either a Python 2 or 3 interpreter, depending on which is given. The default is currently 3. |
| `Debug` | boolean | If the boolean is 1, turns on debug messages and also does not start up a Python instance. |
| `NoInterrupts` | boolean | Turns off interrupts in the interface code. This disables the ability to interrupt running Python code, but makes sure that any interrupts are caught by your own code and not by the interface. |
| `NoDF` | boolean | Turns off automatically setting ⎕DF when importing Python objects. This saves a repr() call (from APL) per object. |

In particular, the following might be of interest:

```apl
py ← ⎕NEW Py.Py('Version' 2) ⍝ use Python 2 instead of 3
```

```apl
⍝ start a Blender instance and control that instead of a normal Python
⍝ (if on Windows, you have to pass in the absolute path to blender.exe instead)
py ← ⎕NEW Py.Py (('PyPath' 'blender') ('ArgFmt' '-P "⍎" -- → ← thread') ('ForceTCP' 1))
```

#### Running Python statements

The `Exec` function can be used to run one or more Python
statements. It takes one string, which may have newlines in it.
The `Py.ScriptFollows` function can be used to help load scripts.

```apl
⍝ run one statement
py.Exec 'import antigravity'

⍝ run a script
py.Exec 'def foobar():',(⎕UCS 10),'  return "abc"'

⍝ or (in a tradfn):
py.Exec #.Py.ScriptFollows
⍝ def foobar():
⍝    return "abc"
```

An `APL` object will be available to the Python code, in order
for it to call back into the APL code. (See [Accessing APL from Python](#accessing-apl-from-python) for more information.)

#### Evaluating Python expressions

The `Eval` function can be used to evaluate a Python expression.
It takes as its left argument the Python expression to be evaluated,
and as its right argument a vector of APL arguments to be substituted
into it. Inside the Python expression, the quad (`⎕`) or the quote-quad
(`⍞`) can be used to refer to these arguments. If the quad is used,
the argument will be converted to a (hopefully) suitable Python
representation first; if the quote-quad is used, the argument will be
exposed on the Python side as an `APLArray` object. (See [Data
conversion](#data-conversion) for more information.)

If the Python expression returns something other than an `APLArray`,
it will be converted back into a suitable APL form before being sent back
to APL.

```apl
     ⍝ access a variable
     py.Eval '__name__'
pynapl.PyEvaluator

⍝ add two numbers
     '⎕+⎕' py.Eval 2 2
4

⍝ this is equivalent to ⍴X
     '⍞.rho' py.Eval ⊂5 5⍴⎕A
5 5

⍝ round trip
     'APL.eval("2+2")' py.Eval ⍬
4

⍝ set a variable on the Python side
     'x' py.Set 42
     py.Eval 'x'
42

⍝ alternate syntax when there are no arguments
     py.Eval 'APL.eval("2+2")' 
4
```

Just as with `Exec`, an `APL` object will be made available to the
Python expression.

#### Making Python functions available to APL

It is also possible to 'import' a Python function to the APL workspace.
The `PyFn` function can be used to create APL functions that call
Python functions automatically.

The `PyFn` function returns a namespace containing two functions,
`Call` and `CallVec`.

* `CallVec` takes a vector of arguments as its
right argument, and passes those into the Python function. It takes an
optional boolean vector as its left argument, which describes
whether or not to convert the arguments.

* `Call` is a "normal" APL function. If used monadically, it calls
the Python function with one argument (`f(⍵)`); if used dyadically
it calls the Python function with two arguments (`f(⍺,⍵)`). The
arguments are always converted.

The namespace also includes a reference to the Py object that created
it, so it will not be destroyed until such functions themselves are.

Example:

```apl

⍝ import a Python module
py.Exec 'import webbrowser'

⍝ define a function from it in APL
⍝ this one handily takes only one argument so can be used monadically
showPage←(py.PyFn 'webbrowser.open').Call

⍝ this will now show a web page
showPage 'http://www.dyalog.com'
```

#### Making Python modules and objects available to APL

By default, Python objects that have APL equivalents are automatically 
converted. E.g., a Python list becomes an APL vector. (See the
"Data Conversion" section.)

Python objects that do not have such equivalents are sent as references
instead, which can be used on the APL side to access their attributes.
On the APL side, a stub class will be instantiated which will have
attributes corresponding to the Python ones.

```apl
      py.Exec'import sys'
      sys←py.Eval'sys'
      sys.version_info
3 5 3  final  0

5↑sys.⎕NL¯2
 __doc__  __name__  __package__  __stderr__  __stdin__ 
```

Fields are exposed on the APL side by means of properties, which can be
used to set and retrieve the values, and methods are exposed as functions
which can be called:

```apl
      os←py.Import'os' ⍝ convenience functions
      +os.getpid ⍬
17906
```

Such functions return a shy result, and take a right argument consisting
of a vector of positional arguments, and an optional left argument representing 
the keyword arguments. This left argument may either be a namespace or a
list of key-value pairs.

```apl
      json←py.Import'json'
      +(⊂'separators' '--')json.dumps ⊂1 2 3 4
[1-2-3-4]
```

It is also possible to send these references back to Python and interact
with them there:

```apl
      '⎕.getpid()' py.Eval os
17906
```

The resulting classes cannot be instantiated from APL using `⎕NEW`, they
can only be instantiated by calling the Python constructors. The objects
keep a reference to the `Py` instance that created them, which means
the Python interpreter will stay alive as long as any of its objects
are still around. On the Python side, the objects are stored by the
interface, and released when all APL references to them have been removed.

In some instances, some name mangling is required. Variables are exposed as
properties on the APL side, which means that a variable `foo` will conflict
with functions named `get_foo` and `set_foo`. In this case, those functions
will be renamed `⍙get_foo` and `⍙set_foo`.

In APL, a Python object reference will have the following members:

- Properties:
    - `foo`: for each non-callable attribute `foo` in the object, gets or sets that attribute
    - `∆foo`: for each non-callable attribute `foo` in the object, gets that attribute
               without object translation (i.e., will _always_ return a Python
               object, even for lists and numbers).
 - Functions:
    - `bar`: for each callable attribute `bar` in the object, calls it and returns the result. 
    - `∆bar`: for each callable attribute `bar` in the object, calls it and returns the result
               without object translation.

- `⍙DF`: returns the string representation of the object (using `repr`), and also sets
              the display form to it.
    - `⍙Get`: retrieves an attribute, given as a character vector as the right argument
    - `⍙Get∆`: retrieves an attribute, given as a character vector as the right argument,
                without object translation on the result.
    - `⍙Set`: sets the attribute (left argument) to the value given as the right argument
    - `⍙SetRaw`: sets the attribute (left argument) to the value given as the right argument,
                  using `⍞` instead of `⎕`.
    - `⍙Call`: calls the function (left argument) with the given positional and keyword arguments
                 (right argument, both must be present).
    - `⍙Call∆`: like `⍙Call`, but without object translation on the result.

#### Error handling

If the Python code raises an exception, the interface will signal a
DOMAIN ERROR. `⎕DMX.Message` will contain the string representation
of the Python exception.

### Accessing APL from Python

The `APL.py` module contains a function that will start an APL
interpreter. Just like the APL side, it expects the `Py.dyalog`
script to be in the same directory.

#### Starting an APL interpreter

An APL object can be obtained using the `APL.APL` function. This
will start a Dyalog instance in the background and connect to it.

```python
from pynapl import APL
apl = APL.APL()
```

An optional `dyalog` argument may be given to the `APL` function,
to specify the path to the `dyalog` interpreter. If it is not given,
on Unix the `dyalog` interpreter on the path will be used,
on Windows the registry will be consulted. 
The Dyalog instance will be shut down once the `apl` object is
destroyed.

#### Fixing an APL script

The `fix` function takes a string, which will be 2⎕FIX'ed on the
APL side. This can be used to load large amounts of APL code into
the interpreter.

```python
apl.fix("""
:Namespace Test
foo←42
:EndNamespace
""")
```

#### Evaluating APL code

The `eval` function takes a string, which will be evaluated
using `⍎` on the APL side. Any extra arguments passed into
`eval` will be put into a vector and exposed as `∆` on the
APL side, and a `py` object will be available for the APL code
to communicate back to the Python interpreter. (See [Accessing
Python from APL](#accessing-python-from-apl).)

This is a relatively low-level function, and it is probably better
to use `fn` and `op`.

Conversion of data *to* APL types is done automatically. (Anything
that's not an `APLArray` is converted.) The result of the evaluation
is converted back to the Python format unless `raw` is set.

```
>>> apl.eval("2+2")
4
>>> apl.eval("⎕A")
u'ABCDEFGHIJKLMNOPQRSTUVWXYZ'
>>> apl.eval("⎕A", raw=True)
<Array.APLArray object at 0x7fb704e36310>
>>> apl.eval("'2+2' py.Eval ⍬") # round trip
4
```

#### Making APL functions available to Python

The `fn` function can be used to import an APL function to Python.
The function may be niladic (if called with no arguments),
monadic (if called with one argument), or dyadic (if called with
two).

As with `eval`, a named argument `raw` can be set to prevent
automatic data conversion.

```
>>> aplsum = apl.fn("+/")
>>> aplsum([1,2,3,4,5])
15
>>> aplsum(3, [1,2,3,4,5])
[6, 9, 12]
```

The function may be an anonymous dfn and may contain newlines.
It may *not* be a definition of a tradfn (those can be defined
using `fix` or `tradfn`, then referred to by name using `fn`).

```python
>>> factorial = apl.fn("""
{ ⍵≤0:1
  ⍵×∇⍵-1
}
""")
>>> factorial(5)
120
```

##### Defining a tradfn using Python

Apart from using `fix`, a tradfn can also be defined using the
`tradfn` function:

```python
>>> foo = apl.tradfn("""
r←foo x
r←x+x
""")
>>> foo(5)
10
>>> apl.fn("foo")(5)
10
```

#### Making APL operators available to Python

Python does not make the difference between functions and
operators that APL makes. Therefore, APL operators can be
exposed as Python functions using the `op` function.

```
>>> scan = apl.op("\\") # note the extra backslash for escaping
```

The operator is then exposed as a Python function, which takes 
one or two arguments, depending on whether the operator is 
monadic or dyadic.

The arguments may be either values or Python functions.
If you want to pass an APL function to an APL operator via Python,
you must first import the APL function using `fn`.

```
>>> apl_add = apl.fn("+")
>>> py_add = lambda x, y: x+y
>>> apl_sumscan = scan(apl_add) # equivalent to "+\"
>>> py_sumscan = scan(py_add)   # uses the Python "+"
>>> apl_sumscan([1,2,3])
[1, 3, 6]
>>> py_sumscan([1,2,3])
[1, 3, 6]
```

If an APL operator is applied to an APL function via Python,
as in the `apl_sumscan` example, this is detected, and the application
is done in APL without calling back into Python.

#### Using APL objects from Python

Just as APL may make use of Python objects, Python may make use of
APL objects.

If a call to `eval` or to an APL function returns an instance of an
APL object, it is represented on the Python side by an instance of the
`APLObject` class, in which public functions and variables will appear
as attributes.

```
>>> apl.fix("""
... :Class foo
...    :Field Public n←0
...    ∇x←getN
...       :Access Public
...       x←n
...    ∇
...    ∇setN x
...       :Access Public
...       n←x
...    ∇
... :EndClass
... """)
['foo']
>>> a = apl.eval("+a ← ⎕NEW foo")
>>> a
<pynapl.Array.APLObject object at 0x7f1d3a9e0ac8>
>>> a.n
0
>>> a.getN()
0
>>> a.setN(20)
[]
>>> a.n
20
>>> a.n = 30
>>> a.getN()
30
```

The `APLObject` class contains a reference to the object on the APL side,
so changes to its state are reflected on the APL side, and vice versa:

```
>>> apl.eval("a.n")
30
>>> apl.eval("a.n←40")
40
>>> a.n
40
```

Members whose names are not valid names in Python can be accessed via `getattr`
and `setattr`. Python 2 requires attribute names to be ASCII only, so members whose
names contain non-ASCII characters cannot be accessed. Python 3 does not have
this limitation.

#### Error handling

If a signal is raised by the APL code, an APLError will be raised
on the Python side. The exception object will contain a `dmx` field,
which is a dictionary that contains the fields from `⎕DMX`.

When an interrupt is raised, the message will be `"Interrupt"` and
`dmx` will be `None`.

### Data conversion

#### From Python to APL

* Numbers (any kind) or boolean: number
 * One-character strings: characters
 * Other string: character vector
 * List or tuple: vector
 * NoneType: empty numeric vector
 * Dictionary: namespace

In addition, any kind of iterable object (objects that are instances
of `collections.Iterable` or that implement `__iter__`, and objects
that implement both `__len__` and `__getitem__`) will be iterated over,
and the results sent as a vector to APL. This allows for most kinds of
custom container objects to be used.

_NOTE_: if the object is an infinite generator, it will cause a hang.

If the numpy library is available, numpy matrices will be automatically
converted to APL matrices.

If the object is none of these, an object reference will be sent to APL,
where it can be used to access its attributes. Python code can also send an
object reference explicitly by using the `apl.obj` function.

```apl
     py.Eval 'sys' ⍝ ask for module object
#.Py.⍙PythonObject.[module]
     py.Eval '[1,2,3,4]' ⍝ send a list
1 2 3 4
     py.Eval 'apl.obj([1,2,3,4])' ⍝ send a list as an object
#.Py.⍙PythonObject.[list]
```

#### From APL to Python

* Numbers: int, long, or float, depending on which fits best
 * Simple (non-nested) character vector: Unicode string
 * Numeric vector / nested vector: List
 * Higher-rank array: nested list (the equivalent of
   `{↓⍣((⊃⍴⍴⍵)-1)⊢⍵}` is done).
 * Namespace containing values: dictionary

#### The `APLArray` class

This is a class on the Python side that can be used to communicate
with APL without going through the conversion. It is a multidimensional
array, which may contain nested APLArray objects.

An APLArray object can be indexed using a list or a tuple.
The index should be given as if `⎕IO=0`, e.g.:

```
>>> foo = apl.eval("5 5⍴⎕A", raw=True)
>>> foo.rho
[5, 5]
>>> foo[2,3]  # ⎕IO←0 ⋄ (5 5⍴⎕A)[2;3]
u'N'
```

Assignment to individual items is possible in the same manner.
Conversion will be done automatically if it is necessary.
An `IndexError` will be raised if the coordinates are out of
range.

## Implementation details

On Unix, there are two ways in which the connection between APL and
Python can be made.

1. The default way is by using two named pipes, which
the initiating side will create (using `mkfifo`) and pass to the
client program.

2. It can also be set to use a TCP connection. The initiating
 side will start up a TCP server (using Conga on the APL side)
 on an unused port, and listen for a connection from the client side.

On Windows, only TCP mode is supported. On Unix, TCP mode may be
necessary to use non-standard interpreters (Blender in particular
does not like pipes much). TCP mode has about twice the latency as
pipe mode.

### Communication

The both programs communicate by sending each other messages,
as described below.

#### Message Format

The underlying format used for messages consists of a 5-byte header
and then a body.

The first byte of the header denotes the message type, the next four
give the length of the body in bytes (high-endian).

The contents of the body are UTF-8 encoded text, usually JSON.

#### Message Types

| Message Type | Contents | Purpose |
| --- | --- | --- |
| `0` (`OK`) | ignored | returned to signal nothing has gone wrong |
| `1` (`PID`) | the PID of the process, as UTF-8 text | sent by the client on startup |
| `2` (`STOP`) | ignored | tells the client to shut down |
| `3` (`REPR`) | an UTF8-encoded string of code | runs the code on the other side and sends `REPRRET` back with the string representation of the result (for debugging) |
| `4` (`EXEC`) | an UTF8-encoded string of code, which does not return a value | runs the code on the other side, and sends back `ERR` or `OK`. For APL this `⎕FIX`es the code |
| `5` (`REPRRET`) | an UTF8-encoded string | sends back the result of an earlier `REPR` |
| `10` (`EVAL`) | a JSON array of two elements, the first being a string of code and the second being an array of serialized objects | evaluates the expression given the arguments, and sends back the result using `EVALRET` |
| `11` (`EVALRET`) | a serialized object | the result of an earlier `EVAL` |
| `253` (`DBGSerializationRoundTrip`) | a serialized object | deserializes and reserializes the object on the other side, then sends the result back using the same message code (for debugging) |
| `255` (`ERR`) | an UTF-8 string containing the description of the error | signal an error |

#### JSON messages

##### `EVAL`:

The `EVAL` message is a JSON list containing two elements. The first element
should be a string containing the expression to evaluate, the second element
should be a (possibly empty) list of arguments.

###### `ERR`:

The `ERR` message is a JSON dictionary containing at least a `Message` field,
which contains the error message. Errors coming from APL may also contain a
`DMX` field, which contains the JSON representation of Dyalog APL's `⎕DMX`
object.

#### Reentrancy

The message handling code on both sides supports handling messages while
waiting for the result of another. E.g., if an `EVAL` is sent, it may
cause another `EVAL` to be sent back, which will then be handled before
the corresponding `EVALRET` is received. This way, the evaluation of an
expression may switch back and forth between the two sides as needed.

Example:

```
Python side:
>>> x = apl.eval("2+'2+2' py.Eval ⍬")
 # Python sends to APL: EVAL '2+2' py.Eval ⍬
 
APL side:
     2+'2+2' py.Eval ⍬
 ⍝ APL sends back to Python: EVAL 2+2

Python side:
>>> 2+2
 # this evaluates to 4
 # Python sends to APL: EVALRET 4

APL side:
 ⍝ receives EVALRET 4
     2 + 4
 ⍝ this evaluates to 6
 ⍝ APL sends to Python: EVALRET 6

Python side:
 # receives EVALRET 6
 # the final answer is 6 
>>> x
6
```

## Testing

To run unit tests from Python:
```sh
$ python2 -m unittest pynapl.test.test_APL pynapl.test.test_Array
$ python3 -m unittest pynapl.test.test_APL pynapl.test.test_Array
```

To run unit tests from APL:
```apl
      ]load pynapl/Py       
      ]load pynapl/test/Test
      Test.RunTests 2 ⍝ test with Python 2
      Test.RunTests 3 ⍝ test with Python 3
```

There is also an interactive GUI test that uses [TkInter](https://wiki.python.org/moin/TkInter):
```apl
      ]load pynapl/Py
      ]load pynapl/PyTest
      PyTest.PyTest
```

pynapl

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README