×

First time here?

You are looking at the most recent posts. You may also want to check out older archives. Please leave a comment, ask a question and consider subscribing to the latest posts via RSS or email. Thank you for visiting!

One of the new features to Silverlight 3 is the ability to add multi-touch capabilities to your application.  When I posed the question on Twitter, I got some responses of ideas people would use this for.  Honestly most of them could be accomplished with mouse events today and X/Y calculations.  These would be the touch applications that are pretty singular.  But I did get some multi-touch ideas that I think I’ll try to explore.  First though, let’s look at the basics of what Silverlight provides for multi-touch application development.

Hardware

Hopefully I’m stating the obvious here, but your hardware has to support multi-touch.  And I’m not talking about that fake kind.  I’m talking about hardware that announces the WM_TOUCH messages to Windows.  If you (or your customers) aren’t going to be having multi-touch hardware, then writing against this API isn’t going to help!  I’m currently using the HP TouchSmart TX2 laptop running Windows 7.  I find this to be a good machine and fairly cheap-ish with regard to how laptops are these days and with the features it provides.

The Event

The first thing to understand is how to tap into the touch events from the hardware to Silverlight.  Understanding this at the beginning of your application development can be a critical step.  The key reason for this is unlike other input events (i.e., MouseLeftButtonDown, etc.) which can be added to individual UIElements in an application, the touch event is an application-wide event. 

There is one primary event: FrameReported.  This event is what gets fired when the hardware sends the touch events to the runtime.  The Touch class is a static class for the sole reason of this FrameReported API.  To wire it up in your application you can use code like this:

   1: Touch.FrameReported += new TouchFrameEventHandler(Touch_FrameReported);

And now you have to write your event handler.

The Event Handler Arguments

Once the runtime receives a touch message, the FrameReported event fires (and will do so several times…see later here).  The arguments you get that you primarily need to concern yourself in most circumstances are the GetPrimaryTouchPoint and GetTouchPoints.

The primary touch point could be thought of as the first touch message/point that the runtime received in a current sequence.  So if your application is using single gesture touch behaviors, this is likely what you’d use.  Otherwise GetTouchPoints is going to give you all the registered touch points from the hardware reported to the runtime.

For me understanding the Move event is critical.  If you take a look and add the data to my diagnostic app below for Move, you’ll see that even simply touching your finger in one place fires constant Move commands.

What you get in a TouchPoint

Both the primary and the collection of touch points listed above will return the TouchPoint object, which contains valuable information.  Namely it is going to give you Postition, which is a point relative to the offset you provided in the GetTouchPoint call (or absolute if you pass in null).

You also get the Action of the touch.  There are three actions: Down, Move and Up.  It is important to understand the firing sequence here.  Assume a given TouchPoint, it will first report Down, then it will continue to report Move until the touch is removed, at which point Up will occur.  The key piece in the middle is Move.  This action is firing even if you aren’t moving any element.  It is essentially reporting that you have a TouchPoint that is in Down state (i.e., touched).  Move can be helpful if you are needing to move things along with the updated position of the element.

You also get the TouchDevice which contains some helpful information.  Provided via the TouchDevice is an Id value, which is a unique id provided by the operating system for the device which reported the TouchPoint.  Also provided is DirectlyOver which is the topmost UIElement the position is over at the time the touch was produced.

What about my mouse events?

Ah, good point!  In the TouchFrameEventArgs you have a method you can call which is SuspendMousePromotionUntilTouchUp.  You would want to use this if you knew *for sure* that the end user had multi-touch hardware.  This would prevent the mouse event promotion for the given touch point.  This method can only be called if the Action is Down for the TouchPoint.  Once the TouchPoints all report Up, then normal mouse event promotion would resume.

Putting it all together

For these basics, I decided just to create a quick diagnostic application that would show the registering of the TouchPoint elements, as well as identifying the primary touch point.  My application has registered the FrameReported event handler and then I’ve added some logic:

   1: public MainPage()
   2: {
   3:     InitializeComponent();
   4:     Touch.FrameReported += new TouchFrameEventHandler(Touch_FrameReported);
   5: }
   6:  
   7: void Touch_FrameReported(object sender, TouchFrameEventArgs e)
   8: {
   9:     foreach (TouchPoint tp in e.GetTouchPoints(this.Positions))
  10:     {
  11:         if (tp.Action == TouchAction.Down)
  12:         {
  13:             // do something
  14:         }
  15:     }
  16: }

The end result is that when the user touches that application surface, we add the TouchPoint to an ObservableCollection that is bound to a DataGrid to show the points currently registered and by which device.  When the user removes the touch action, they go away.

Obviously it is hard to demonstrate touch capabilities in a screenshot and it really does it no justice.  I’m going to do my best attempt here to show you a picture-in-picture view of the application running and me interacting with it via Touch.  You’ll need Silverlight to view this demonstration.

Summary

There you have it.  The basics of multi-touch in Silverlight 3.  It’s fairly simple to understand the core mechanics of the API.  What gets tricky is interacting with your application beyond just showing the points :-).  In a future post I’ll show an application that makes use of this multi-touch feature in understanding where the touch occurred in my given application and how you can find the element that was touched (even though it’s an application-wide event).  If you aren’t subscribed, please consider subscribing to my blog for regular updates for Silverlight information.

Feel free to spelunk this diagnostic app code:  MultiTouch_CS.zip (C#) or MultiTouch_VB.zip (Visual Basic).

Hope this helps!


This work is licensed under a Creative Commons Attribution By license.


7/30/2009 2:25 PM | # Cool Stuff Tim!
Really cool stuff there Tim. I can see a lot of uses there for kiosks and really any program that can benefit from touch input where WPF might be overkill.
7/30/2009 2:30 PM | # re: Silverlight 3 Multi-touch: The Basics
Nice article, what if any significance is there in the values for point.TouchDevice.Id. On my TouchSmart TX2 the values are 10, 12, 13 and 14.
7/30/2009 2:45 PM | # re: Silverlight 3 Multi-touch: The Basics
Kevin - mine register 10,12,13,4 -- I'm assuming these translate to the hardware device mappings or unique identifiers in the hardware...Silverlight just receives what the hardware sends. My understanding is they map to the touch sensors.
7/30/2009 3:45 PM | # re: Silverlight 3 Multi-touch: The Basics
Not to be an ingrate here, but what is the market share of MultiTouch enabled devices? That the Silverlight team decided to focus on this, as opposed to web cam support, right click support, additional 3d support, a rich text editor, a selectable textblock, a flow document container, a native menu control, etc, etc * 1000, indicates a complete break down in product management. Unless this was an extraordinarily quick feature to implement, this is one that clearly should have been put off until at least version 4.
7/30/2009 4:11 PM | # re: Silverlight 3 Multi-touch: The Basics
Jack -- INGRATE!!! ;-) It's a fair question. When you look at the resources and tasks of what it takes for features, there is a balance between scheduling of course. The touch support in Silverlight is a port of what was there for Surface, etc. -- so most was there already. You pointed out a lot of things that are being looked at for Silverlight 4. I wouldn't say it is a breakdown in product management unless you are acutely aware of the product roadmap *and the reasons for that roadmap.* I'm not saying it is right/wrong...just that it is easier for us all to say 'you should have...' when we aren't in charge of managing those resources, localization needs, test resources, etc. Touch provides Silverlight with a competitive advantage in the field and with the growth of touch applications I'd say it was important. Just as important are the features you pointed out and we're working hard to address them!
7/30/2009 4:12 PM | # re: Silverlight 3 Multi-touch: The Basics
Cool stuff Tim, multi touch rocks!
I've just started a codeplex project to play with it, check out http://multitouch.codeplex.com :)
7/30/2009 4:20 PM | # re: Silverlight 3 Multi-touch: The Basics
@Jack - Multi touch is good for mobile applications, although Silverlight is not fully available for mobile phone yet.
7/30/2009 11:03 PM | # re: Silverlight 3 Multi-touch: The Basics
Cool post, Tim.
I'm wondering what version of Windows and N-Trig driver did you install for this demo?
The wind told me Touch Driver wouldn't work with Windows 7 RTM 7600.16385 on TX2.
As far as I know, the version of N-Trig multi-touch driver was later than 2.59.
7/31/2009 12:14 AM | # re: Silverlight 3 Multi-touch: The Basics
Well done, as usual Tim.
Do you now if there is a list of compatible tablets for multi touch? And if so, where can I find it?
Not that I don't trust your judgement on the TX2, but I'd like to compare for myself.
Gravatar
7/31/2009 7:08 AM | # re: Silverlight 3 Multi-touch: The Basics
Super - I hope this is useful on multi-touch WinMo 7 devices (hint hint!). One question - WPF provides Gesture events with data like Zoom, Pan, etc. Do we get that in SL too?
7/31/2009 8:47 AM | # re: Silverlight 3 Multi-touch: The Basics
Gongdo - I'm temporarily using the 2.59 drivers until the release ones get signed.

Jonathan - I'm not sure what touch devices are out there for laptops other than the Dell XT2 and the TouchSmart TX2. I believe Toshiba has one too, but I haven't investigated. I suspect maybe the Windows 7 product site might have some insight.

JC -- no native gesture support. You can do your own translation, but Silverlight doesn't get the WM_GESTURE messages.
7/31/2009 8:56 AM | # re: Silverlight 3 Multi-touch: The Basics
Very cool! Will the same approach work on WPF apps? Do you have any information on supporting multi-touch with WPF? Thanks!
7/31/2009 12:10 PM | # re: Silverlight 3 Multi-touch: The Basics
VERY cool stuff!!!
8/1/2009 10:34 AM | # re: Silverlight 3 Multi-touch: The Basics
Jeff,
Thanks for the excellent info, but another Ingrate here... It seems that MS Product Management thought that spinning, bouncing cubes, rather than usable data grids with proper binding, would give Silverlight a "marketing edge". And, now, they think that support of multi-touch without gestures (which are essential to actual use of a touch-screen app) and without such basics as right-click give it another "edge". Well, what it does is cement Silverlight's "glitz/hype over functionality/usefulness" reputation.

I can't figure out how to use this basic multi-touch stuff without writing a lot of code, most of which will become redundant and obsolete (and will probably break) when those features _do_ get released from Microsoft. And some of which will probably violate multi-touch UI conventions -- e.g., how to distinguish a "touch and hold" from a flick or rotate. So, even if we want to, how are we supposed to justify using this version of Silverlight for multi-touch? Any thoughts on how or when some of these basics will get provided prior to the full Silverlight 4 release? Maybe along with .NET 4, when other support for multi-touch in WPF comes along?

P.S.-I can understand how some of this functionality will be needed by Microsoft for further development work, and how getting it into the hands of non-MS developers ASAP can be a good thing. But unless their roadmap is made public, "official" release of this kind of stuff just adds to confusion and thrash.

Thanks again for posting this info, tho! -- Thomas
8/3/2009 3:45 PM | # re: Silverlight 3 Multi-touch: The Basics
Thanks for the very nice and detailed post, I just wrote a simple app and it doesn't accept Touch inputs while I am FullScreen mode. Is it for security or bug?
8/3/2009 4:36 PM | # re: Silverlight 3 Multi-touch: The Basics
Jobi -- it is not enabled for right now. There is a work item to look at enabling it. I've requested the docs to be updated to make that more clear. What's the full-screen scenario you have that wouldn't work in IE kiosk mode?
8/3/2009 6:46 PM | # re: Silverlight 3 Multi-touch: The Basics
Thanks Tim for clarifying this. Yep the IE Kiosk mode will work well for me.
8/5/2009 7:48 PM | # re: Silverlight 3 Multi-touch: The Basics
Gestures support (and more) for Silverlight 3 multitouch:

http://nuigroup.com/forums/viewthread/6591/

Regards,
Gene.
8/7/2009 11:29 AM | # re: Silverlight 3 Multi-touch: The Basics
Yes! multitouch... it Rocks.. i bought a TX2 specially to upgraded to vista... will have to play aound with Silverlight to see the full scale of the multitouch... im not gonna sleep toniht just thinking of the applications :D Great Post!
8/14/2009 12:57 PM | # re: Silverlight 3 Multi-touch: The Basics
Use MultiTouchVista (http://multitouchvista.codeplex.com) to emulate multitouch hardware in Windows 7.
8/18/2009 8:57 PM | # re: Silverlight 3 Multi-touch: The Basics
Cool stuff Tim!. U r awesome dude!. I really like this new stuff. But my problem is I dont have a multi touch device to play around. is there any software that can enable my laptop to be like multi touch feature? (by using 2 mouse)
:)

keep it up Tim.
8/22/2009 9:07 AM | # re: Silverlight 3 Multi-touch: The Basics
@Daniel D
I'm currently using Windows 7 only with VirtualBox. Do you think, MultiTouchVista will work inside the virtual environment (2 mice simulating MultiTouch)?
10/20/2009 5:58 PM | # re: Silverlight 3 Multi-touch: The Basics
Is TouchDevice.Id the best way to track an individual touch throughout it's life? (in order to differentiate between different fingers moving around on the screen). It seems to work and create a new ID per finger and not reuse that ID until that finger is released.
10/21/2009 1:26 AM | # re: Silverlight 3 Multi-touch: The Basics
Tim - the device ID is reported by the hardware -- so keep that in mind. It is only as reliable as what the hardware tells us the touch points are registering. Given that though, I've had the same experience as you -- as long as the touchpoint is in Down still, the ID doesn't change.
10/30/2009 3:17 PM | # re: Silverlight 3 Multi-touch: The Basics
Just doing a little post, my client is so desperate for Silverlight developers, please let me know if you want a contract job. Silverlight is the new IT. Tim i have been following your advice through out my technical career. It helps me in screening candidates, you are a awesome dude
11/4/2009 5:41 AM | # re: Silverlight 3 Multi-touch: The Basics
But ... what about a way to tell if are running the application on a multi-touch hardware or on a regular screen, not a touchscreen ?
11/4/2009 8:03 AM | # re: Silverlight 3 Multi-touch: The Basics
Alex -- if you get TouchPoints -- you're on multi-touch hardware :-) -- You can examine the DeviceType as well to see.
11/5/2009 12:52 AM | # re: Silverlight 3 Multi-touch: The Basics
I want to render a different interface when the user has a touch screen; so I can't wait for the user to tap the screen (or ask him to do so), then, if I get a response in the Touch.FrameReported handler, think what to display.
11/10/2009 3:40 AM | # re: Silverlight 3 Multi-touch: The Basics
It's interesting to take a look at the architecture of the recently released opensource MT4J library (MultiTouch for Java), that has Gesture processors and abstraction layers for various hardware (including the TUIO protocol and using multiple mice as input for easier development/debugging without using multitouch hardware-in-the-loop). Could follow similar MT modelling in Silverlight I think
11/19/2009 11:20 PM | # re: Silverlight 3 Multi-touch: The Basics
hello. thanks for sharing this. one question: where to find a multi touch laptop?
11/20/2009 7:59 AM | # re: Silverlight 3 Multi-touch: The Basics
Sameer - there are MANY available now from Dell, HP, Acer, etc. I have an HP TouchSmart TX2
2/2/2010 6:58 PM | # re: Silverlight 3 Multi-touch: The Basics
I have a HP TX2 laptop, but it only recognizes 2 fingers, how do you enabled more than 2 fingers touch ?
Thanks
Gravatar
3/10/2010 12:46 PM | # re: Silverlight 3 Multi-touch: The Basics
Im also wondering how you got it to recognize more than 2 touchpoints on the tx2. I heard that by default, the drivers the tx2 comes with only supports 2 touchpoints. Did you install another driver? If so which one?
3/10/2010 1:16 PM | # re: Silverlight 3 Multi-touch: The Basics
Get your drivers from NTrig
2/21/2011 6:00 AM | # re: Silverlight 3 Multi-touch: The Basics
The end aftereffect is that if the user touches that appliance surface, pass4sure 70-505 we add the TouchPoint to an ObservableCollection that is apprenticed to a DataGrid to appearance the credibility currently registered and by which device. pass4sure 70-528 If the user removes the blow action, pass4sure 70-620 they go away.
4/20/2011 12:09 AM | # re: Silverlight 3 Multi-touch: The Basics
The end result is that when the user touches that application surface, we add the TouchPoint to an ObservableCollection that is bound to a DataGrid to show the points currently registered and by which device. When the user removes the touch action, they go away. k visit ad visit am visit u visit
4/21/2011 11:28 PM | # re: Silverlight 3 Multi-touch: The Basics
The end result is that when the user touches that application surface, we add the TouchPoint to an ObservableCollection that is bound to a DataGrid to show the points currently registered and by which device. When the user removes the touch action, they go away. v visit
4/24/2011 1:35 AM | # re: Silverlight 3 Multi-touch: The Basics
These would be the touch applications that are pretty singular. But I did get some multi-touch ideas that I think I’ll try to explore. w visit
3/30/2012 1:42 PM | # re: Silverlight 3 Multi-touch: The Basics
Tim, please, can you explane to me, whats the magic remove ellipsises in the sourse project? Can't understand how this instruction can translate tp.TouchDevice.Id to UIElement (Ellipse):
Positions.Children.Remove((UIElement)this.FindName(tp.TouchDevice.Id.ToString()));

The tp.TouchDevice.Id.ToString() always return 10 for one mice and 100 for another one, how thats strings could be cast to UIElement ?

 
Please add 2 and 4 and type the answer here:

DISCLAIMER:

The opinions/content expressed on this blog are provided "ASIS" with no warranties and are my own personal opinions/content (unless otherwise noted) and do not represent my employer's view in any way.