.. _listening-and-connecting: Listening and Connecting ************************ In the fundamental networking scenario there is a server listening at a known address, a client connecting to that same address and agreed mechanisms for the presentation of requests and the receiving of responses. Traditionally these interactions are synchronous in nature where the client submits the request and waits for a response from the server. There can be additional code required to get application data on and off the wire, activity known as marshaling and encoding. This library supports that traditional style of networking but also goes much further. By adopting the asynchronous model of operation from `ansar-create `_ the library manages to provide a much more flexible and sophisticated style of networking, while also hiding most of the technical details behind the messaging system that is built into async operation. Features and behaviours that were difficult or impossible to implement with traditional networking become achievable. Other benefits include code clarity and maintainability. Download materials supporting this guide and setup the environment with the following commands; .. code:: $ cd $ git clone https://github.com/mr-ansar/listening-and-connecting.git $ cd listening-and-connecting $ python3 -m venv .env $ source .env/bin/activate $ pip3 install ansar-connect A Complete, Minimal Server ========================== Implementation of a server looks like; .. literalinclude:: include/listen-at-address.py :language: python :tab-width: 4 The essential points in this server are; * the call to :func:`~.create_object` to create an instance of ``listen_at_address()``, * the call to :func:`~.listen` that establishes the object as a service at the configured network address, * the first call to :meth:`~.Buffering.select` to verify the :func:`~.listen`, * the message loop that waits for the next async event, * the call to :meth:`~.Point.reply` that sends the response after receiving a ``Hello()`` message, * the ``return ar.Aborted()`` statement that terminates the server on a control-c. .. note:: This arrangement of an application is inherited from `ansar-create `_. It takes care of persistent settings, provides logging and manages platform signals such as control-c. The :meth:`~.Point.reply` method is a shorthand for :meth:`~.Point.send`. It passes the current *return address* as the destination, ie. the address the ``Hello()`` message came from. Everything to do with networking such as sockets and their related operations, is fully integrated into async operation. The same :meth:`~.Point.send` method used to transfer messages between async objects within a process, is also used to transfer messages across network transports. A command to run the server looks like; .. code:: $ cd <../listening-and-connecting> $ source .env/bin/activate $ python3 listen-at-address.py --debug-level=DEBUG A Matching Client ================= The matching client looks like; .. literalinclude:: include/connect-to-address.py :language: python :tab-width: 4 The essential points in this client are; * the call to :func:`~.create_object` to create an instance of ``connect_to_address()``, * the call to :func:`~.connect` to initiate a network connection, * the call to :meth:`~.select` to confirm the connection, * the call to :meth:`~.Point.send` in response to a :class:`~.socketry.Connected` message, * the second call to :meth:`~.select` in expectation of a ``Welcome()`` message, * the ``return welcome`` statement that terminates the client. .. _defining-a-set-of-reusable-messages: Defining A Set Of Reusable Messages =================================== It is a good idea to create a separate file for message definitions. These are the messages that will be exchanged by the client and server. This serves a similar purpose to a header or include file from other languages. In this context they are referred to as *interface files*; .. literalinclude:: include/hello_welcome.py :language: python :tab-width: 4 The essential points in the ``hello_welcome.py`` module are; * import of ``ansar.encode`` reflecting the singular purpose of the module, * definition of the ``Hello()`` and ``Welcome()`` classes, * registration of each message class using the :ref:`bind() ` function, * passing explicit type information in the ``SCHEMA`` table. The module path and filename are used to auto-generate a message identity at the moment of registration (i.e. the call to :ref:`bind() `). Message definitions can appear in the module in which they are used. However, creation of interface files is a good defense against circular import problems. A Complete Request-Response Sequence ==================================== A run of the client appears below (some header details omitted for clarity); .. code:: $ cd <../listening-and-connecting> $ source .env/bin/activate $ python3 connect-to-address.py -dl=DEBUG 20:32:21.639 + <0000000a>SocketSelect - Created by <00000001> 20:32:21.639 < <0000000a>SocketSelect - Received Start from <00000001> 20:32:21.639 > <0000000a>SocketSelect - Sent SocketChannel to <00000001> 20:32:21.639 + <0000000b>PubSub[INITIAL] - Created by <00000001> 20:32:21.640 < <0000000b>PubSub[INITIAL] - Received Start from <00000001> 20:32:21.640 + <0000000c>object_vector - Created by <00000001> 20:32:21.640 ~ <0000000c>object_vector - Executable "/home/buster/connect-to-address.py" as object process (2363848) 20:32:21.640 ~ <0000000c>object_vector - Working folder "/home/buster" 20:32:21.640 ~ <0000000c>object_vector - Running object "__main__.connect_to_address" 20:32:21.640 ~ <0000000c>object_vector - Class threads (3) "subscribed" (3),"published" (2),"connect-to-peer" (1) 20:32:21.640 + <0000000d>connect_to_address - Created by <0000000c> 20:32:21.640 + <0000000e>SocketProxy[INITIAL] - Created by <0000000a> 20:32:21.640 ~ <0000000a>SocketSelect - Connected to "127.0.0.1:32011", at local address "127.0.0.1:37560" 20:32:21.640 > <0000000a>SocketSelect - Forward Connected to <0000000d> (from <0000000e>) 20:32:21.640 < <0000000e>SocketProxy[INITIAL] - Received Start from <0000000a> 20:32:21.640 < <0000000d>connect_to_address - Received Connected from <0000000e> 20:32:21.640 > <0000000d>connect_to_address - Sent Hello to <0000000e> 20:32:21.640 > <0000000d>connect_to_address - Sent StartTimer to <00000003> 20:32:21.640 < <0000000e>SocketProxy[NORMAL] - Received Hello from <0000000d> 20:32:21.641 > <0000000a>SocketSelect - Forward Welcome to <0000000d> (from <0000000e>) 20:32:21.641 > <0000000d>connect_to_address - Sent CancelTimer to <00000003> 20:32:21.641 < <0000000d>connect_to_address - Received Welcome from <0000000e> 20:32:21.641 ^ <0000000d>connect_to_address - At client - Hello "Gladys", my name is "Buster" 20:32:21.641 X <0000000d>connect_to_address - Destroyed 20:32:21.641 < <0000000c>object_vector - Received Completed from <0000000d> 20:32:21.641 X <0000000c>object_vector - Destroyed 20:32:21.740 < <0000000b>PubSub[NORMAL] - Received Stop from <00000001> 20:32:21.740 X <0000000b>PubSub[NORMAL] - Destroyed 20:32:21.741 > <0000000a>SocketSelect - Sent Close to <0000000e> 20:32:21.741 < <0000000e>SocketProxy[NORMAL] - Received Close from <0000000a> 20:32:21.742 X <0000000e>SocketProxy[NORMAL] - Destroyed 20:32:21.742 > <0000000a>SocketSelect - Forward Closed to <0000000d> (from <0000000e>) 20:32:21.743 X <0000000a>SocketSelect - Destroyed Welcome + your_name: "Gladys" + my_name: "Buster" $ Messages are encoded and decoded automatically using the `ansar-encode `_ library. A full set of types is supported including generic containers (e.g. arrays, vectors and sets), user-defined types and graphs. .. note:: Generation of logs is automatic. Adding a ``--debug-level`` argument directs the logs to ``stderr``, else they are discarded. The return value of the object created by :func:`~.create_object` is rendered on ``stdout``. These behaviours are inherited from `ansar-create `_ and can be controlled in different ways. Network Connections As Asynchronous Addresses ============================================= The logs include information relating to the async object called :class:`~.socketry.SocketProxy`. This object is the gateway between the world of async messages and the world of network sockets. Each network connection involves two proxy objects - one at the connecting end and one at the accepting end. When a client receives the :class:`~.socketry.Connected` message or the server receives the :class:`~.socketry.Accepted` message they were sent from proxy objects. The *return address* (i.e. ``self.return_address``) is then the proxy - the local representation of a remote object. That address can be passed around for use by other parts of the application. The result of sending to a proxy is always the transfer of the message across the underlying connection to the remote end. Once a connection is established there is the ability to perform the familiar request-response operations found in traditional networking. However, with **ansar-connect** there is no real constraint on who can send what and when they can send it. Network connections can be treated as a *multiplexed transports*. Every message transferred across a connection has address information tacked on such that the receiver is always able to direct a message back to the correct sender, e.g. the ``connect_to_address()`` and ``listen_at_address()`` objects. Connections can support multiple ongoing conversations. Activity like this is safe even in the presence of multi-threading. Separation Of Session And Application ===================================== Placing network details behind an abstraction such as async addresses is a useful strategy in several ways. The less technical details that appear in an implementation the more clearly the application is manifested in the source code. Another strategy available in the library to help with code clarity, is the separation of session management and application messages. The former includes messages like :class:`~.socketry.Connected`, :class:`~.socketry.Accepted` and :class:`~.socketry.Abandoned` while the latter includes all the messages that might be exchanged between clients and servers once a connection is established. This issue is most obviously apparent at the listening end, where the server must be prepared to receive :class:`~.socketry.Accepted` and :class:`~.socketry.Abandoned` messages intermingled with application messages like ``Hello()``. All the session management messages and all the application messages *for every connected client* are being directed to a single point. .. note:: The files ``listen-sessions-at-address.py`` and ``connect-session-to-address.py`` are part of the repo downloaded at the start of this section. They are functional equivalents of ``listen-at-address.py`` and ``connect-to-address.py``, respectively. Running the session-based implementations uses the same commands except for the different module names. Clients and servers are inter-changeable, e.g. ``connect-to-address.py`` can be used to connect to ``listen-sessions-at-address.py``. A session-based server looks like; .. literalinclude:: include/listen-sessions-at-address.py :language: python :tab-width: 4 The essential points in this server are; * the addition of the ``accepted_at_address()`` function, * the use of CreateFrame() to capture all the details of a session object, * passing the session variable to the :func:`~.listen` function to enable sessions in the network machinery. An instance of ``accepted_at_address()`` is created after every successful connection of a client. This simplifies the coding of ``listen_at_address()`` - with no duties to perform around the comings and goings of clients, related processing has been stripped out. It also creates a dedicated context for each connected client. As the goals of the application become more ambitious the correct handling of individual clients becomes more complicated and less practical to implement at a single point, i.e. ``listen_at_address()``. A session-based client looks like; .. literalinclude:: include/connect-session-to-address.py :language: python :tab-width: 4 The essential points in this client are; * the addition of the ``connected_to_address()`` function, * the use of CreateFrame() to capture all the details of a session object, * passing the session variable to the :func:`~.connect` function to enable sessions in the network machinery. An instance of ``connected_to_address()`` is created after a successful connection. There is an exchange of ``Hello()`` and ``Welcome()`` messages and the client session object terminates. This causes the shutdown of the underlying socket and the sending of an :class:`~.socketry.Abandoned` message within the server. The demand for client-side session objects is less obvious. The ``connect_to_address()`` function served as a context for the single connection supported by the :func:`~.connect` function. However, the separation still improves code clarity and there is also the ability to move the session objects around. As a slightly odd example it would be possible to switch the client and server session objects around so that the server made a request to the client. It is also possible to combine multiple session object pairs over a single connection. Effectively this would run multiple exchanges side-by-side. Async objects such as ``listen_at_address()`` and ``connect_to_address()`` are known as network *controllers* and async objects such as ``accepted_at_address()`` and ``connected_to_address()`` are known as network *sessions*. Session objects follow a standard object lifecycle - there is a start, an exchange of messages and a termination. There is nothing in that cycle that involves technical networking details. Messages associated with termination of a session object are mapped onto appropriate networking messages, e.g. a :class:`~.lifecycle.Completed` message becomes a :class:`~.socketry.Closed` message sent to the controller. Due to the different routing of session vs application messages the session objects do not receive :class:`~.socketry.Connected` or :class:`~.socketry.Accepted` messages and therefore miss out on the implicit availability of proxy addresses. For this reason the addresses for the proxy and their associated controller are passed as named variables at the time a session object is created, e.g. ``remote_address``. When sessions are enabled the :class:`~.socketry.Connected` and :class:`~.socketry.Accepted` messages are still sent to the appropriate controllers but the return address changes to the address of the newly created session object. By this mechanism controllers have timely visibility of their sessions. .. _networking-with-finite-state-machines: Networking With Finite State Machines ===================================== Finite state machines are particularly well-matched to the requirements of more sophisticated networking. An example would be a software solution involving multiple components spread around a network, e.g. a laboratory with many complex scientific devices or a production line with a high level of automation. Rather than multiple clients making RPC-style requests to a server, there are peer processes making connections within the group and exchanging bi-directional messages over extended periods. It is the ongoing messages that maintain a relationship between the two peers and it is the set of relationships that give the software solution its collective behaviour. In this documentation, examples of async objects such as clients and servers are implemented as functions. This is for reasons such as readability and the typically smaller number of source lines required. It is worth noting that the implementation of **ansar-connect** involves over 100 async objects and almost 100% of this number are implemented as finite state machines. Function-based async objects have their place but the more ambitious networking components are likely to be implemented as FSMs. .. note:: The files ``listen-fsm-at-address.py`` and ``connect-fsm-to-address.py`` are part of the repo downloaded at the start of this section. They are functional equivalents of ``listen-at-address.py`` and ``connect-to-address.py``, respectively. Running the FSM-based implementations uses the same commands except for the different module names. Clients and servers are inter-changeable, e.g. ``connect-to-address.py`` can be used to connect to ``listen-fsm-at-address.py``. Finite state machines are a part of the `ansar-create `_ library. For details on FSMs and a broader understanding of asynchronous programming, refer to the associated documentation. The FSM version of the server looks like; .. literalinclude:: include/listen-fsm-at-address.py :language: python :tab-width: 4 The salient points in this server are; * replacement of the ``listen_at_address()`` function with the ``ListenAtAddress`` class, * inclusion of :class:`~.machine.StateMachine` as a base class, * passing of ``INITIAL`` to the base __init__() call, * the set of functions with names like ``ListenAtAddress_PENDING_Stop``, * registration of the ``ListenAtAddress`` class using :ref:`bind() `, * passing the ``LISTEN_AT_ADDRESS_DISPATCH`` dictionary to the registration, * passing ``ListenAtAddress`` to :func:`~.create_object` Functionally this implementation is equivalent to previous versions. However, operationally this version of the server is *lighter* in that there is no dedicated thread, allowing large numbers of servers to be created without stressing the local computing resources. While a large number of server objects in a process seems unlikely, this property of FSMs has utility. It also has an improved chance at robustness and reliability courtesy of the formal nature of FSMs. The FSM version of the client looks like; .. literalinclude:: include/connect-fsm-to-address.py :language: python :tab-width: 4 The session and FSM capabilities can be combined. Functions such as ``accepted_at_address()`` and ``connected_to_address()`` would become the ``AcceptedAtAddress`` and ``ConnectedToAddress`` classes. It is also possible to mix the implementation styles. Implementation of these variants is left as an exercise. .. _exchanging-messages-with-multiple-servers: Exchanging Messages With Multiple Servers ========================================= Implementing a software component that requires the services of multiple servers is a good coding challenge. Complexity rises quickly as the number of servers increases. There is the initial establishment of all the required connections to consider and then the need to implement recovery after the loss of a connection. Recovery will likely involve multiple attempts to connect and periods of delay to avoid floods of connections. In yet more complex scenarios there may be the ongoing demands of upstream clients to manage. At some inconvenient moment in the development of this component there might be a moment of insight - every connection can be lost at any moment and this is true from the moment a networking component starts through to the moment it terminates. The component may start, acquire a partial set of connections and then lose them all. It is much less stressful - and more accurate - to think of the component as an item of software that is continually attempting to establish a full set of connections. The component is always aware of its current state but achieving a full set is just one state of many and as transient as the others. To simplify their implementation some components may choose to give the fully-connected state special significance, e.g. by responding to its upstream clients with *temporarily-out-of-service* in any other state. .. note:: The file ``connect-and-reconnect-to-address.py`` is included in the repo downloaded at the start of this section. It is a functional equivalent of ``connect-to-address.py``. Consider the following revision of the client; .. literalinclude:: include/connect-and-reconnect-to-address.py :language: python :tab-width: 4 The salient moments in this client are; * the use of :class:`~.grouping.GroupTable` to define a list of servers (i.e. currently one), * the call to :meth:`~.grouping.GroupTable.create` to launch a *connection manager*, * the processing of messages such as :class:`~.grouping.GroupUpdate` and :class:`~.lifecycle.Ready`, * the separate ``say_hello()`` function to allow for cleanup of the connection manager. The ``connect_to_address()`` function has delegated the responsibility for connecting to a connection manager. The first step is to define a list of the desired connections using :class:`~.grouping.GroupTable`. The :meth:`~.grouping.GroupTable.create` method is then used to start the manager which initiates all connection activity and then provides a stream of updates to the object specified at creation time (i.e. the passing of ``self``). Each time a new connection is established or an existing connection is lost, a :class:`~.grouping.GroupUpdate` is sent. In addition the ``Ready`` and ``NotReady`` messages are sent to demarcate a period where all connections are present. The :class:`~.grouping.GroupUpdate` messages must be passed to the :meth:`~.grouping.GroupTable.update` method. This is the mechanism by which the ``group`` attributes reflect the current status of the members listed in the :class:`~.grouping.GroupTable`, e.g. the ``group.server`` value is set to the address of a proxy object. To observe an example of the work carried out by the connection manager (:class:`~.grouping.AddressGroup`), start the client before the server. The client will start a series of connection attempts that succeeds once the server has started; .. code:: $ cd <../listening-and-connecting> $ source .env/bin/activate $ python3 connect-and-reconnect-to-address.py -dl=DEBUG 20:17:17.496 + <0000000a>SocketSelect - Created by <00000001> 20:17:17.496 < <0000000a>SocketSelect - Received Start from <00000001> 20:17:17.496 > <0000000a>SocketSelect - Sent SocketChannel to <00000001> 20:17:17.496 + <0000000b>PubSub[INITIAL] - Created by <00000001> 20:17:17.496 < <0000000b>PubSub[INITIAL] - Received Start from <00000001> 20:17:17.496 + <0000000c>object_vector - Created by <00000001> 20:17:17.496 ~ <0000000c>object_vector - Executable "/home/buster/connect-and-reconnect-to-address.py" as object process (2408477) 20:17:17.496 ~ <0000000c>object_vector - Working folder "/home/buster" 20:17:17.496 ~ <0000000c>object_vector - Running object "__main__.connect_to_address" 20:17:17.496 ~ <0000000c>object_vector - Class threads (3) "subscribed" (3),"published" (2),"connect-to-peer" (1) 20:17:17.496 + <0000000d>connect_to_address - Created by <0000000c> 20:17:17.496 + <0000000e>AddressGroup[INITIAL] - Created by <0000000d> 20:17:17.496 < <0000000e>AddressGroup[INITIAL] - Received Start from <0000000d> 20:17:17.496 + <0000000f>ConnectToAddress[INITIAL] - Created by <0000000e> 20:17:17.496 < <0000000f>ConnectToAddress[INITIAL] - Received Start from <0000000e> 20:17:17.497 > <0000000a>SocketSelect - Sent NotConnected to <0000000f> 20:17:17.497 < <0000000f>ConnectToAddress[PENDING] - Received NotConnected from <0000000a> 20:17:17.497 > <0000000f>ConnectToAddress[PENDING] - Sent StartTimer to <00000003> 20:17:19.498 < <0000000f>ConnectToAddress[GLARING] - Received GlareTimer from <00000003> 20:17:19.499 > <0000000a>SocketSelect - Sent NotConnected to <0000000f> 20:17:19.499 < <0000000f>ConnectToAddress[PENDING] - Received NotConnected from <0000000a> 20:17:19.499 > <0000000f>ConnectToAddress[PENDING] - Sent StartTimer to <00000003> 20:17:24.005 < <0000000f>ConnectToAddress[GLARING] - Received GlareTimer from <00000003> 20:17:24.006 + <00000010>SocketProxy[INITIAL] - Created by <0000000a> 20:17:24.006 ~ <0000000a>SocketSelect - Connected to "127.0.0.1:32011", at local address "127.0.0.1:46152" 20:17:24.006 > <0000000a>SocketSelect - Forward Connected to <0000000f> (from <00000010>) 20:17:24.007 < <00000010>SocketProxy[INITIAL] - Received Start from <0000000a> 20:17:24.007 < <0000000f>ConnectToAddress[PENDING] - Received Connected from <00000010> 20:17:24.007 > <0000000f>ConnectToAddress[PENDING] - Sent UseAddress to <0000000e> 20:17:24.007 < <0000000e>AddressGroup[PENDING] - Received UseAddress from <0000000f> 20:17:24.007 > <0000000e>AddressGroup[PENDING] - Sent GroupUpdate to <0000000d> 20:17:24.007 > <0000000e>AddressGroup[PENDING] - Sent Ready to <0000000d> 20:17:24.008 < <0000000d>connect_to_address - Received GroupUpdate from <0000000e> 20:17:24.008 < <0000000d>connect_to_address - Received Ready from <0000000e> 20:17:24.008 > <0000000d>connect_to_address - Sent Hello to <00000010> 20:17:24.008 > <0000000d>connect_to_address - Sent StartTimer to <00000003> 20:17:24.008 < <00000010>SocketProxy[NORMAL] - Received Hello from <0000000d> 20:17:24.013 > <0000000a>SocketSelect - Forward Welcome to <0000000d> (from <00000010>) 20:17:24.014 > <0000000d>connect_to_address - Sent CancelTimer to <00000003> 20:17:24.014 < <0000000d>connect_to_address - Received Welcome from <00000010> 20:17:24.014 ^ <0000000d>connect_to_address - At client - Hello "Gladys", my name is "Buster" .. Adding a server is now trivial. Just create another CreateFrame() and add it to the list of named parameters for :class:`~.grouping.GroupTable`. The new name will be added to the ``group`` and a value assigned via the :class:`~.grouping.GroupUpdate` message and :meth:`~.grouping.GroupTable.update` mechanism. Initially all the named attributes of the ``group`` object are set to ``None``. Control flow through the client is arranged to reach the *ready* state. The assigment of a proxy address with the ``server_address = group.server`` statement occurs after the latest message has been checked against the :class:`~.lifecycle.Ready` type which results in a ``break`` from the ``while`` loop. Minor changes to the flow could produce a client that runs forever, always attempting to maintain a full set of connections. There is a lot more going on under the hood of the manager. Delays between connection attempts include *backoff* and *randomization*. Each successive delay is longer (up to a maximum) and also tweaked by a random amount. These are efforts to reduce noise on the network and also to avoid swamping a server immediately after it joins the fray. There are also distinct delay sequences for the different scopes of network - attempts to connect across the loopback interface occur with shorter delays than attempts to connect across the LAN. Attempts to connect across the Internet are separated by the longest delays. The :class:`~.grouping.GroupTable` works with several different types of entry. As well as :class:`~.networking.ConnectToAddress` there is also :class:`~.networking.ListenAtAddress` and several more. Together these cater to just about every arrangement of networking components and inter-component connections that might be needed. Encryption ========== Security is a part of networking. For this reason encryption is built into the **ansar-connect** library. To activate encryption there is a simple boolean flag on both the :func:`~.listen` and the :func:`~.connect` functions; .. code:: python def listen(self, requested_ipp, session=None, encrypted=False): .. .. def connect(self, requested_ipp, session=None, encrypted=False): .. .. For networking to succeed the values assigned at each end must match. Either they are both ``True`` or they are both ``False``. Encryption is based around Curve25519 high-speed elliptic curve cryptography. There is an initial Diffie-Hellman style exchange of public keys in cleartext, after which all data frames passing across the network are encrypted. All key creation and runtime encryption/decryption is performed by the `Salt `_ library. Authentication and authorization is left to the user.