DroneController

In this post I will talk about an extension I added using the .NET DroneController. The extension itself is based on the new WorkflowDesigner control available in the Workflow Foundation 4.

In order to see a working example download the either Wilke.Interactive.Drone.Simple or Wilke.Interactive.Drone.PlayGround from codeplex.

It is now much easier to re-host the WorkflowDesigner and allow users to compose workflows in a custom application. This is the feature I want to apply in the Path Designer. You can find much more on re-hosting the WorkflowDesigner on the web, a starting place is here.

In order to provide a design experience to the user I created custom activities with corresponding activity designers. Using these activities a user can now compose a new workflow that will the ARDrone fly according to predefined commands.

The PathDesigner van be shown as a modal window as follows:

if (droneController != null && droneController.ConnectionStatus == ConnectionStatus.Open)
{
inputProvider = new AssistInputProvider(droneController);
inputProvider.Initialize(droneController.RollThrottleValue, droneController.PitchThrottleValue, droneController.HeightThrottleValue, droneController.YawThrottleValue);

PathDesigner pathDesigner = new PathDesigner(droneController);
pathDesigner.ShowDialog();

//Switch back to the selected inputprovider
inputProvider = new WiimoteInputProvider(droneController);
inputProvider.Initialize(droneController.RollThrottleValue, droneController.PitchThrottleValue, droneController.HeightThrottleValue, droneController.YawThrottleValue);
}
else
{
MessageBox.Show("This feature is only available if a connection is made with the ARDrone.");
}

From the code above you can see the following:

  • The PathDesigner needs an active instance of an object that implements the IDroneCommander interface (e.g. an active DroneController instance)
  • When working with the path designer I create an instance of the AssistInputProvider. This is a variation of the WiimoteInputProvider, the difference is that there is no accelerometer feedback. I use this input provider to make adjustments during the automated flight and in case of emergency I can land the ARDrone without having to run to the laptop.

This is how the UI looks like:

qiaywbow

On the left you see the activities that are currently supported. The middle panel shows the design surface. It is here that you can drag and drop activities to compose your workflow. Once the workflow has been created you can save it for later reuse. Clicking on the run button will start the workflow and pass the commands to the ARDrone.

There is also a possibility to activate workflow tracking. When activated the workflow engine will provide visual feedback on the current activity with a yellow border. There is still an issue with tracking in case of parallel activities. I will talk a bit more on this in another post were I will go into a bit more detail on some programming aspects of the DroneController and its related projects.

Here is a small video that shows the basic process:

Small demo on using the PathDesigner with the .NET DroneController

Remark:

The PathDesigner is experimental, I only implemented just enough to demonstrate the concept. Since I do not have that much space in my living room I was not able to play with it and design more advanced paths. Let me know if you like the concept.

In the last post I will take up some aspects of the DroneController and its related projects in somewhat deeper detail.

The binaries for this application can be found at http://dronecontroller.codeplex.com/releases. This application uses some new functionalities of WPF 4.0.

The interface of the application looks as follows:

nnpoejb2

The application uses a DroneController instance that connects to an ARDrone with networkid ‘ardrone_229’. I f you would like to get this application up and running make sure you replace this value in the code where the connection with the ARDrone is made.

Note: This image is taken in evening light conditions in my living room. It also gives a slight idea of the available surface I had working most of the time with the ARDrone. In the process I ruined at least two of my plants that are located in the corner of the living room.

As I already indicated the application is designed for a wide screen multi touch screen (1920×1080). On the right side of the screen you can see two controls that allow to pilot the ARDrone by multi-touch gestures on the screen. The radar allows you to control roll and pitch, it also supports single finger rotation which results in changing the yaw parameter of the ARDrone. The joystick on the left of the radar allows you to control height and yaw of the ARDrone. The button with the cog on top of the joystick allows to switch the joystick with two slider controls that can also be manipulated with gestures.

h4glbaex

The UI also lets you change the device you use to control the ARDrone:

image

Here you see the drop down menu that allows you to change the input provider. The Sensor Board is a circuit board that contains several sensors (accelerometer, touch, ambient light) that can be used to demonstrate the new Sensor capabilities available on Windows 7. I f you do not own such a circuit board this inputprovider can of course not be used. This is a link to the product page of the Flexis JM Badge Board.

For the design I based myself on some resources I found on the web:

Remark: The name of this application is PlayGround, and that is what it is. I am not a professional designer (although I would like to be somewhat more proficient on this), so this application is just to experiment with a couple of things. Just as I learned a lot from looking around on the web I am sure this application also has something to offer on this level. I always am eager to learn so if you find things that can be considered an ‘insult’ to professional coding please let me know.

In the following post I will talk about the Path Designer. This is an experimental extension that uses a re-hosted workflow designer (Windows Workflow Foundation 4).

In this post I will talk in a bit more detail how to create a new application that uses the .NET DroneController. The application I create will be a WPF application mostly because WPF has native support for WritebleBitmap. If we would use a windows forms application we would need to find a way to bind the WritebleBitmap to a PictureBox control. I did not yet spend time doing this.

When creating a new application there are basically three different phases to take care off:

  • Implement a method that creates a new DroneController instance, configures it and connects it to the ARDrone.
  • Implement methods that pass commands via the DroneController instance to the ARDrone.
  • Implement event handlers that allow to work with data coming from the ARDrone.

Let’s go over each of these phases in more detail:

Create a new DroneController instance:

When you create a new instance of the DroneController class you pass an instance of a DroneControllerConfiguration class. This can be done explicitly or implicitly.

Suppose I want to create a new instance of the DroneController using all the defaults I can do just:

DroneController droneController = new DroneController();

In this case the DroneController instance will be configured using the defaults of the DroneControllerConfiguration class:

/// <summary>
/// Initializes a new instance of the <see cref=”DroneControllerConfiguration”/> class.
/// </summary>
public DroneControllerConfiguration()
{
this.NetworkIdentifier = “ardrone_229”;
this.DroneIpAddress = “192.168.1.1”;
this.GuestIpAddress = “192.168.1.2”;
this.NavigationDataPort = 5554;
this.VideoStreamPort = 5555;
this.CommandPort = 5556;
this.ControlInfoPort = 5559;
this.EnableATCommandThread = true;
this.EnableNavigationDataThread = true;
this.EnableVideoStreamThread = true;
this.EnableControlInfoThread = false;
this.EnableInputFeedback = false;

this.RollThrottle = Constants.DefaultRollThrottleValue;
this.PitchThrottle = Constants.DefaultPitchThrottleValue;
this.HeightThrottle = Constants.DefaultHeightThrottleValue;
this.YawThrottle = Constants.DefaultYawThrottleValue;

DroneInfoTimerInterval = Constants.DroneInfoTimerInterval;
}

Here you see that the network identifier of the adhoc network created by the ARDrone is called ‘ardrone_229’. Yours will most probably be different, in this case you can change the default constructor of the DroneControllerConfiguration class.

Another way would be to construct a DroneController instance as follows:

DroneController droneController = new DroneController(“[your network identifier]”);

Here the DroneController instance will be created with all default values but

If you want to have more control you can create first an instance of a DroneControllerConfiguration class and pass this on when you create a new instance of the DroneController.

droneControllerConfiguration = new DroneControllerConfiguration();

droneControllerConfiguration.NetworkIdentifier = “ardrone_229”;
droneControllerConfiguration.EnableNavigationDataThread = true;
droneControllerConfiguration.EnableVideoStreamThread = true;
droneControllerConfiguration.EnableATCommandThread = true;
droneControllerConfiguration.EnableControlInfoThread = false;
droneControllerConfiguration.VideoFilePath = videosPath;
droneControllerConfiguration.PictureFilePath = picturesPath;
droneControllerConfiguration.EnableATCommandSimulation = simulate;
droneControllerConfiguration.EnableInputFeedback = true;

#region Throttle Values

droneControllerConfiguration.RollThrottle = .2f;
droneControllerConfiguration.PitchThrottle = .2f;
droneControllerConfiguration.YawThrottle = .6f;
droneControllerConfiguration.HeightThrottle = .4f;

#endregion

droneController = new DroneController(droneControllerConfiguration);

Remark:

I included the concept of throttle values because not everyone has a large empty warehouse at their disposition, in my case I only had +/- 2 m2 in my living room to spare. By playing with the throttle values you can minimize the effect of your input values on the ARDrone movement so that you can use and experiment with it in smaller places. Although this is also possible by changing some parameters on the ARDrone itself I decided to provide more accessible support with these throttle parameters on the DroneControllerConfiguration instance.

In order to be able to monitor the DroneController we can subscribe to the different events that this class exposes:

droneController.TraceNotificationLevel = TraceNotificationLevel.Verbose;
droneController.OnNotifyTraceMessage += new EventHandler<TraceNotificationEventArgs>(droneController_OnNotifyTraceMessage);
droneController.OnNotifyVideoMessage += new EventHandler<VideoNotificationEventArgs>(droneController_OnNotifyVideoMessage);
droneController.OnNotifyDroneInfoMessage += new EventHandler<DroneInfoNotificationEventArgs>(droneController_OnNotifyDroneInfoMessage);
droneController.OnConnectionStatusChanged += new EventHandler<ConnectionStatusChangedEventArgs>(droneController_OnConnectionStatusChanged);

Now we can connect to the ARDrone:

droneController.Connect();

Pass commands to the ARDrone

After you have created an instance of a DroneController class, it knows how to communicate with the ARDrone, you only have to tell it what commands it should send. This can be done in two ways:

Commands send directly to the DroneController instance

e.g.

if (droneController != null && droneController.ConnectionStatus == ConnectionStatus.Open)
{
droneController.SetFlatTrim();
droneController.StartEngines();
}
Commands send via an InputProvider

e.g.

if (droneController != null && droneController.ConnectionStatus == ConnectionStatus.Open)
{
inputProvider = new WiimoteInputProvider(droneController);
inputProvider.Initialize(droneController.RollThrottleValue, droneController.PitchThrottleValue, droneController.HeightThrottleValue, droneController.YawThrottleValue);
}

In this case no explicit commands are send to the DroneController instance. The object that implements the IInputProvider interface consumes the methods exposed by the IDroneCommander interface.

Implement the DroneController event handlers

The DroneController class exposes several events that provide feedback on different aspects. It is not mandatory to subscribe to these events but it makes working with the DroneController a lot easier.

These are the different events that are exposed:

  • OnNotifyTraceMessage – This event is used for general diagnostic messages. The TraceNotificationLevel enumeration can be used to throttle the level of detail.
  • OnNotifyVideoMessage – This event conveys the current video frame captured by the ARDrone cameras. The composition of the video frame depends on the selected video channel.
  • OnNotifyDroneInfoMessage – This event provides feedback on several aspects of the ARDrone, e.g. Battery Level. The existing code can be adapted if more information is needed.
  • OnConnectionStatusChanged – This event provides connection status feedback.
  • OnNotifyInputMessage – This event provides feedback on changes in flight parameters. This can be used to provide visual feedback on screen.
  • OnVisionDetectMessage – This event is triggered when tag detection is activated, it will tell whether a tag is detected an provide information on the detected tag.

Do not forget that the event handlers are not activated on the UI thread, if you need to access controls from within the event handler you need to go via the dispatcher (look in the sample source code how to do this if you are not familiar with this concept).

In the next post I will talk a small bit about the more advanced multi-touch WPF interface.

In this post I will talk about how to use the .NET DroneController with the ARDrone. The DroneController is created with Visual Studio 2010 and targets the .NET 3.5 framework. The main reason I target this framework is because the DroneController uses the updated WritebleBitmap class, this class exposes a BackBuffer property which allows modifying the image while is currently rendered and avoid smearing or tearing.

All code has been developed on a Windows 7 platform, using the code on Windows XP or Vista should just be fine (the only thing I am not sure about is the video encoding part, I have no clue what default functionality is available on these additional platforms). In case of issues on these part I can always take a look.

Running simple WPF client

Download the latest binaries from http://dronecontroller.codeplex.com/releases and extract to your hard disk.

Before you connect to the ARDrone make sure

  • The batteries are connected and all the green led’s are activated.
  • Your WI-FI adapter has a fixed IP address of 192.168.1.2 (this is the default address given by the ARDrone DHCP, since there is an issue on Windows platform with the ARDrone DHCP server behavior we make it a fixed address in stead of a dynamic)
  • You have connected to the adhoc network created by the ARDrone

If all goes well you should see following window:

00alzxq3

Change the Network Id with the name of the adhoc network created by your ARDrone.

Remark: Recording video and taking pictures relies on the existence of a C:Temp directory. Either make sure this directory exists or change the source code.

As you can see the interface is really simple. There are a couple of buttons and sliders to control the ARDrone. There are also two output areas, one for showing the video and another one for showing trace messages.

If you see messages like Nav:No activity on worker thread for 500 milliseconds. it means that the thread servicing the Navigation Data has timed out. Just disconnect and reconnect again.

Remark

The flight parameters you send to the ARDrone to change roll, pith, height or yaw have a value between –1 and 1. This simple interface has limited these parameters between –.1 and .1 in order to avoid making too brisk maneuvers. You can always adapt these settings if you have enough room.

In the next post I will discuss a bit more the ins and outs of creating a consumer for the .NET DroneController.

Finally I found some time to post my experiences with implementing a library in native C# that allows to pilot the ARDrone. In stead of explaining what the ARDrone exactly is I like just to point to the product site itself which can be found at: http://ardrone.parrot.com/parrot-ar-drone/usa/.

It basically is a programmable remotely controlled quadricopter. Communication happens via Wi-Fi. The out of the box API support focuses on development on iPhone, although it is possible to take the existing C SDK and compile it for use on other platforms. You can find plenty of examples when you look on the web. I took a slightly different approach in that I did not use the existing SDK (except for research on how it all works) but created a completely managed C# library from scratch.

This is the first post of the following series of articles:

The purpose of this article is not to go into much detail on the actual code, this is an effort I will maybe take up gradually at a later time, but to stay pragmatic and explain how the library can be used in real life.

This is a picture that shows the high level composition of the .NET DroneController:

image

The important parts of the DroneController library are:

DroneControllerConfiguration: When a new DroneController is created you can use the DroneControllerConfiguration to configure it.

Communication Center: This is the heart of the DroneController, it provides the actual communication between the internals of the DroneController and the ARDrone. The Communication Center spawns for each communication channel a new thread on which it sets up a UDP or TCP session. Currently four communication channels are supported:

  • The Navigation Data channel: This channel receives information from the ARDrone like status, positional parameters, tag detection info and so on. (UDP)
  • The VideoStream channel: This channel receives the raw data coming from the two video cameras. (UDP)
  • The ATCommands channel: This channel is used to send the commands to the ARDrone. (UDP)
  • The Control channel: This channel is use to send/receive control information to and from the ARDrone. (TCP)

Command Center: This is the place where commands are sent to. These commands are formatted according to the rules defined by the ARDrone and put on a command queue. The Communication Center empties this queue on a predefined interval.

DroneProxy: This represents the ARDrone in the DroneController. Information about the ARDrone is stored at this level. Getting information from the DroneProxy is like getting information from the actual ARDrone.

IDroneCommander: This is the contract that allows consumers to interact with the DroneController. It defines the methods supported by the ARDrone like: Takeoff, Land, SteFlatTrim, PlayLedAnimation and so on. The DroneController implements this contract.

IInputProvider: This is the contract that has to be implemented by device controllers. The contract defines an entry point for the IDroneCommander. In this way can the device controller pass commands to the DroneController.

What does the DroneController support:

  • Pilot the ARDrone via the supported commands (look for further info on the official developer support site of Parrot)
  • Make video recordings of the frames captured by the ARDrone cameras. The output format is WMV.
  • Save the current video frame as pictures. The output format is JPEG.
  • Subscribe to several events that provide vital information (video, connection status, health status, ARDrone info, Tag detection info, …)
  • Several input providers are supported including Wiimote. Additional ones can be easily created by implementing the IInputProvider interface.

What’s in the related projects:

  • A sample of a simple WPF interface using the DroneController.
  • A more advance sample using multitouch features of .NET 4.0 and Windows 7.
  • An experimental extension that uses a Workflow metaphor to design automated flights.
  • Experimental follow me procedure that makes the ARDrone follow a coloured tag (this is something that still needs some redesign and refinement)

The DroneController is written completely in C#, the video recording and wlan connectivity part in it and the Wilke.Interactive.Drone.InputProviders assembly rely on following third party libraries:

This is the version info found on the ARDrone (telnet 192.168.1.1 and then type cat /data/config.ini) :

  • num_version_config = 1
  • num_version_mb = 17
  • num_version_soft = 1.0.4
  • soft_build_date= 2010-07-16 15:12

I did not update to the latest firmware since I read on the forum about some instabilities. This might or might not be the case but I did want to avoid extra complications. According to the latest official documentation no major changes have taken place on the level of AT Commands between the two last firmware updates so the DroneController should work fine with it. If people using the latest firmware do have issues please let me know.

In the next part I will talk about how to use the ARDrone controller.