SDK Tools Guides

Home

Writing Extensions for MoSync

We encourage everyone to help us develop MoSync, and writing new extensions for MoSync is probably the most common thing you will want to do. In this topic we introduce you to some important core concepts that you need to understand and lead you step-by-step through creating an API extension.

This topic assumes that you have a good understanding of how to use MoSync, that you understand the build process, and that you can build the MoRE runtime in Visual Studio. It also assumes that you have some basic familiarity with the MoSync code.

Core Concepts

In this section we introduce some core concepts that you will need to understand when you write extensions.

Runtimes and user code

Interaction of user code and runtime through syscalls

User code includes all the code that you write when you create an application with MoSync along with some of the MoSync libraries, such as the MAUI library.

A runtime is the thing that executes the user code on the device. For each platform that MoSync supports, there is a different runtime. Some runtimes are written in C++, others like the J2ME and Android runtimes are written in Java. Each runtime has two parts: one that executes the user code and one that interfaces to it to provide low-level services.

Syscalls and events

Communication between the user code and a runtime is through syscalls and events. Syscalls are like the traditional calls a user application makes to an OS kernel. Whenever the user code needs to do something it cannot do on its own, such as creating a socket connection, it performs a syscall to the runtime. To create a socket connection, for example, the syscall is maConnect.

A syscall is an actual instruction which the runtime traps during code execution. The instruction only has one argument, and that is the syscall number. A syscall, like any ordinary function call, can return a value. A syscall can do pretty much any thing you'd like it to do, but what it can never do is to block. If the syscall cannot finish immediately, such as when waiting for socket data, it needs to return and have the blocking call run in a thread in the background. Once it has finished, it will return the result through events which the user fetches through yet another syscall called maGetEvent.

The maIOCtl syscall

The maIOCtl syscall is a special syscall which is meant for extensions which aren't supported or fully supported on all the MoSync runtimes. What this means in practice is that anything that isn't part of the standard API, that is anything that isn't the least common denominator on all platforms,  will be implemented through an maIOCtl. Implementing an MaIOCtl is simpler than implementing an entirely new syscall, and is what you will be using to implement your extensions. The only limitation that the maIOCtl syscall has is that it only takes three parameters. But on the bright side, any or all the three can be user defined structures, which means that you can always pass all the parameters you like in a structure.

maapi.idl and Pipe-Tool

Any syscall starts its life in a file called maapi.idl. Here all syscalls, events, and event types (among other things) are defined. To minimize the amount of code redundancy, we have created a tool called idl2 which takes the maapi.idl file and generates several important files:

  • $MOSYNCDIR/bin/asm_config.lst — the file used by Pipe-Tool to recognise syscalls
  • runtimes/java/shared/invoke_syscall_java.h the inner part of switch case which handles different syscalls for the java runtimes
  • runtimes/cpp/core/invoke_syscall_cpp.h the inner part of switch case which handles different syscalls for the c++ runtimes
  • runtimes/java/shared/generated/core_consts.h opcodes and such for java runtimes
  • runtimes/java/shared/generated/maapi_consts.h constants in the maapi.idl file
  • intlibs/helpers/cpp_defs.h constants and structs in the maapi.idl file, for the C++ runtimes
  • intlibs/helpers/cpp_maapi.h all the syscall declarations for the C++ runtimes
The idl2 tool also generates a set of files with the string "_IX_" in their filename. These are extensions that have been implemented through the maIOCtl syscall. To the MoSync GCC, a syscall is the same as just a normal function call. It is completely unaware of the concept syscall. Recognising syscalls and patching the binary code with the syscall opcode is the job of pipetool. The way it recognises the syscalls is quite simple, it looks at the generated asm_config file to see if a function call is actually a syscall or not. If it is, then its call opcode is replaced with a syscall opcode.

The maapi.idl format is quite simple. It contains declarations for syscalls in a C-like syntax:
RETURN_TYPE IDENTIFIER ( (in|out) TYPE IDENTIFIER, ...);
where:
  • RETURN_TYPEis 
    • void [*]
    • [unsigned] char [*]
    • [unsigned] short [*]
    • [unsigned] int [*]
    • [unsigned] long [*]
    • [unsigned] float [*]
    • [unsigned] double [*]
  • IDENTIFIER is a normal C identifier (for example, maConnect )
  • in|out - Tells the idl2 tool whether the parameter goes in or out.
  • TYPE is any primitive or user defined type
A syscall declaration looks like this:
int maFrameBufferGetInfo ( out MAFrameBufferInfo info );
Note that the declarations are written in between:
#if IX_SOMETHING . . . #end 

For maIOCtl declarations the block ends up in a separate file with a similar name. It is in this block that you will be writing API extensions. The idl2 tool also produces wrapper code for any extensions that have been written through maIOCtl. This is another one of the advantages of the idl2 tool. It takes care of all the little details and updates every where that needs to be updated. The wrapper code might look something like this (from libs/MAStd/IX_CELLID.h):

static inline int maGetCellInfo ( MACellInfo* pInfo ) {     return maIOCtl( 56, (int)pInfo, 0, 0 ); } 

Creating an Extension Step-by-Step

Now that the concepts are in place, it's time to get our hands dirty. There are five steps to writing an API extension:
  1. Edit tools/idl2/maapi.idl
  2. Run the idl2 tool
  3. Add your implementation to the runtimes you need
  4. Rebuild all the user libraries
  5. Try it out

In this example we will write a simple extension as an maIOCtl, we will pass a structure with 16 chars in it, and will return the same chars through an event.

Step 1 — Editing tools/idl2/maapi.idl

We need to make four changes in this file:

  • Add an extension define
  • Add a new event type
  • Add a new inner structure to the event structure
  • Add a new maIOCtl

Adding the extension define

Edit tools/idl2/extensions.h file and add to it "#define IX_TUTORIAL". This is needed to process anything that is declared between '#if ...' and '#endif' in the maapi.idl file. Here is the start of our example:

#define IX_RESOURCE_TYPES
#define IX_GUIDO
#define IX_WLAN
#define IX_FILE
#define IX_RECORD
#define IX_CELLID
#define IX_CALL
#define IX_STREAMING
#define IX_CONNSERVER
#define IX_OPENGL_ES
#define IX_AUDIOBUFFER
#define IX_SEGMENTED_DATA
#define IX_PIM
#define IX_TUTORIAL

Adding the new event type

The MAEvent structure contains a field called "type", this field is an integer which identifies the event type. So next we need to add a constant which identifies our custom event. At the end of the block called "constset int EVENT_TYPE_" we add our custom event type:

#if IX_TUTORIAL
        TUTORIAL = 10001;
#endif

Note: To avoid collisions with future extensions from the MoSync team, we suggest that you always choose a large value for the event constant.

In the version of maapi.idl that we are working with, the entire block looks like this:

constset int EVENT_TYPE_ {
        /**
        * This event is posted when the operating system sends MoSync a command 
          * to exit. Causes include the OS shutting down and OS-controlled user 
         * commands. 
         * \see maGetEvent()
        */
        CLOSE = 1;

        KEY_PRESSED = 2;
        KEY_RELEASED = 3;
        /// Connection
        CONN = 4;
        /// Bluetooth discovery
        BT = 5;
#if IX_GUIDO
        /// Has MAEvent::key be the identifier for the TTS session, as returned 
        /// by maStartSpeaking().
        TTS = 6;
#endif    //IX_GUIDO
#if IX_WLAN
        /// Uses MAEvent::state.
        WLAN = 7;
#endif    //IX_WLAN

        POINTER_PRESSED = 8;
        POINTER_RELEASED = 9;
        POINTER_DRAGGED = 10;

#if IX_CALL
        /// Has MAEvent::state be one of the \link CALLSTATE \endlink constants.
        CALL = 11;
#endif    //IX_CALL

        /**
         * While MoSync doesn't have focus, no key events will arrive and the 
          * screen will not be updated. If the keypad is locked, no application 
          * will have focus.
         * \see maLockKeypad
         */
        FOCUS_LOST = 13;
        FOCUS_GAINED = 14;

#if IX_STREAMING
        /**
        * Has MAEvent::data point to a MAStreamEventData.
        */
        STREAM = 15;
#endif    //IX_STREAMING
//#if IX_LOCATION
        /// Has MAEvent::data point to an MALocation.
        LOCATION = 16;

        /// MAEvent::state is one of the \link #MA_LPS_AVAILABLE MA_LPS 
          /// \endlink constants.
        LOCATION_PROVIDER = 17;
//#endif    //IX_LOCATION
#if IX_AUDIOBUFFER
        /// MAEvent::state is TEMPLATE_PAGE_CONTENTgt; 0 when the audio stream is waiting for more data,
        /// or TEMPLATE_PAGE_CONTENTlt; 0 on error.
        AUDIOBUFFER_FILL = 18;
#endif
        SCREEN_CHANGED = 21;

#if IX_TUTORIAL
         /// Just a simple event type added for the tutorial
         TUTORIAL = 10001;
#endif
    }

Adding the new inner structure to the event structure

Now that we have a new event type, it is time to create a way through which we can return data to the user from an event. This can be accomplished in two ways, one is to allocate dynamic memory in the runtime and let the runtime copy that memory to user memory. The runtime will also update the data pointer in the MAEvent struct. To do this, you have to write a few macros in the right places. The second method is to create a new struct with the data that you need to return to the user and embed this in the MAEvent struct. The second method has the advantage of being a bit simpler, but at the cost of bloating the MAEvent struct. For the sake of simplicity, we will only go through the later method in this document. So let's get started.

First, we define our struct. We can call it MATutorialEvenData, it needs to contain 16 chars. The next step is to embed it in the MAEvent struct, in the revision of the code base that the author is working with, the entire code will look like this.

struct MATutorialEventData {
    char m_data[16];
}

struct MAEvent {
    /**
    * One of the \link #EVENT_TYPE_CLOSE EVENT_TYPE 
    * \endlink constants.
    */
    int type;
    union {
        struct {
            /**
            * In KEY events, this will be one of the 
            * \link #MAK_UNKNOWN MAK \endlink constants.
            */
            int key;
            /**
            * In KEY events, this will be the native keycode.
            */
            int nativeKey;
        } ked;

        /**
         * Tutorial structure
         */
        MATutorialEvenData tutorial;

        /**
         * In POINTER events, this will be the location of the pointer.
         */
        MAPoint2d point;

        /**
        * In \link #EVENT_TYPE_BT BT \endlink events, this will be a value TEMPLATE_PAGE_CONTENTgt;= 0 or
        * one of the \link #CONNERR_GENERIC CONNERR \endlink constants.
        */
        int state;

        /**
        * Valid only in CONN events.
        */
        MAConnEventData conn;

        /**
        * Used by custom events. See invididual event descriptions.
        */
        void* data;
    }
}

Adding the new maIOCtl

 

At this point we have all the parts in place to return data through events, now we just need a way of making a call to the runtime. This is where syscalls and maIOCtl comes in. We will define a new IOCtl in the maIOCtl block (after the opening bracket and before the closing bracket). The maIOCtl will look like this:

#if IX_TUTORIAL
  int maSyscallTutorial(in MAString data);
#endif //IX_TUTORIAL

where MAString is a defined as 'typedef char* MAString' at the top of the maapi.idl file.

Step 2 — Running the idl2 tool

Now that we've made actual changes to the syscall interface, we need to regenerate all the auto generated files that have to do with syscalls. In Visual Studio, build tools/idl2 and run it. Your output should look like this,

If you instead get this:

D:\workspace\repo-new\MoSync2.3\tools\idl2>Debug\idl2.exe
Error opening file "opengl_generated.idl" in
maapi.idl on line 2383
Exception: Unknown exception

Then you need to build the opengl_generated.idl, run tools\GLWrapperGenerator\build.bat and run the idl2 tool again.

You probably noticed that for most files it generates three different versions of it. For instance, the IX_TUTORIAL extension goes in:

  • intlibs\CPP_IX_TUTORIAL.h
  • libs\MAStd\IX_TUTORIAL.h
  • runtimes\java\shared\generated\IX_TUTORIAL_consts.h
The first one is to be used from the C++ runtimes (MoRE, Symbian, Windows mobile, Linux). The second file is to be used on the user side. The last file is to be used from the Java runtimes (such as J2ME and Android).
So let's look inside the first two files ( since we're only doing the changes for a C++ runtime in this example). In the first file we have:
#ifndef IX_TUTORIAL_DEFS_H #define IX_TUTORIAL_DEFS_H /** \brief A hash of the abstract representation of the API described in this file. * Identifiers, declarations and definitions are included in the calculation of the hash, * but documentation is not. */ #define MAIDL_HASH ((int)0xc2fcb710) /// Just a simple event type added for the tutorial #define EVENT_TYPE_TUTORIAL 10001 #define maIOCtl_maSyscallTutorial 247 #endif //IX_TUTORIAL_DEFS_H

In here we have hash (which is discussed in section 4), the event ID, and syscall ID. We will need these to write the code which handles the maIOCtl call in the runtime. The java version contains similar things.

In the next file we have:

#ifndef IX_TUTORIAL_H
#define IX_TUTORIAL_H

#ifdef __cplusplus
extern "C" {
#endif

/// Just a simple event type added for the tutorial
#define EVENT_TYPE_TUTORIAL 10001

static inline int maSyscallTutorial(const char* data) {
    return maIOCtl(247, (int)data, 0, 0);
}

#ifdef __cplusplus
}
#endif

#endif    //IX_TUTORIAL_H

As you see, this file is a bit different, there is an actual C function in there, this is because we wrote our new syscall/extension in the maIOCtl block. The idl2 tool knows that anything inside that block is not an actual syscall, but goes through the maIOCtl syscall. The first parameter is the maIOCtl ID, and in the previous file you can see that the preproccesor define "maIOCtl_maSyscallTutorial" is set to 247, which is quite practical and the reason that the previously discussed hash is needed to make sure that both the user libraries and the runtime agree on the same syscall interface.

Step 3 — Add your implementation in the runtimes you need

 

By this point, we have all the parts in place to do the maIOCtl on the user side.  Now you need to write the code on the runtime side which actually handles the IOCtl. In this document, the only runtime we will modify is MoRE, but modifying the other runtimes is similar.

In the source directory, there exists a file runtime/cpp/platforms/sdl/SyscallImpl.cpp, this file handles all the syscalls in the SDL runtime (MoRE). In the Visual Studio project you can find it under Platforms/cpp/sdl/SyscallImpl.cpp. Open it, and add the include "#include <helpers/CPP_IX_TUTORIAL.h>", this will include the definitions that have to do with our example.

Next we need to add the actual code for handling the IOCtl. So, scroll down to the part that says:

SYSCALL(int, maIOCtl(int function, int a, int b, int c)) {

This is the actual maIOCtl syscall. In it you will find a big switch statement. We need to add our own handler to this switch case. So just before the default case, we add this piece of code:

        case maIOCtl_maSyscallTutorial:
        {
            char *  data;
            MAEvent myevent;
            
            // Get pointer to memory
            data = (char*)SYSCALL_THIS->GetValidatedMemRange( a, 16 );

            // Copy to event
            myevent.type = EVENT_TYPE_TUTORIAL;
            memcpy( myevent.tutorial.m_data, data, 16 );

            // Put in event queue
            gEventFifo.put( myevent );

            // Return success code
            return 1;   
        }            

Let's go through the parts of this code. Firstly, a pointer to the argument that was passed to the maIOCtl from the user side, is fetched. The method GetValidatedMemRange takes a pointer to the user side memory and the number of bytes that the pointer points to. If the memory is valid, a pointer will be returned. Otherwise a panic will occur.  Secondly, the event type is set and the ASCIIZ string passed by the user is copied to the event structure. Thirdly, the event is put in the event queue, from where it can be fetched by the user through the maGetEvent syscall. And lastly, we return from the maIOCtl syscall.

Assuming that you have Visual Studio correctly set up and you can build MoRE, rebuild MoRE. The executable will be copied to your MoSync installation path (MOSYNCDIR/bin).

Step 4 — Rebuilding all the user libraries

This step is very important. Any change you make to the IDL file might change the binary interface of syscalls (specfically syscall IDs). Because of this there is a hash in the generated maapi.h file that is verified against the runtime to make sure the user code and the runtime have been built with the same interface. So we need to remove all the old files and rebuild the libs:

  1. In your MoSync installation dir (let's call it MOSYNCDIR), delete include/*
  2. delete MOSYNCDIR/lib/pipe/*
  3. In the source directory (let's call it MOSYNCSRC), do cd libs
  4. ruby workfile.rb
  5. ruby workfile.rb CONFIG=""

Step 5 — Trying it out

If you have gotten everything to work this far, there should be only one last step remaining, and that is to actually test that it works as expected. In the MoSync IDE, create a project and create in it a file called main.c containing the following code:

#include <ma.h>
#include <IX_TUTORIAL.h>

int MAMain ( )
{
    while ( 1 )
    {
        MAEvent e;
        while ( maGetEvent( &e ) != 0 )
        {
            switch ( e.type )
            {
                case EVENT_TYPE_CLOSE:
                    return 0;

                case EVENT_TYPE_KEY_PRESSED:
                case EVENT_TYPE_POINTER_PRESSED:
                    maSyscallTutorial( "world!" );
                    break;

                case EVENT_TYPE_TUTORIAL:
                    printf( "Hello %s\n", e.tutorial.m_data );
                    break;
            }
        }
    }

    return 0;
}

If you've done everything correctly, this code will compile and run just fine. And in that case, congratulations, you've just written your first MoSync extension!

MoSync SDK 3.3
Copyright © 2013 MoSync AB
www.mosync.com