Using C# in Batch files

Batch Files

People usually smile when I say that some parts of our network like the filtering of system events are held together by batch files. It just seems so arcane, but there are some big benefits:

You don’t have to compile anything, you always know where the source code is and you can simply copy them between machines without having to set up anything. And since it’s more or less a legacy technology, Microsoft isn’t really changing a lot anymore, so there’s little chance of an upgrade breaking a batch file.

The only problem is: cmd.exe shell syntax is a horrible, horrible mess and even the most basic string functions can take ages to implement… plus, the code you’ll write look like gibberish to anybody else no matter what you do. Plus there’s the horrible, horrible string escaping behavior and the very strange behavior of variables.

 

PowerShell

So, Microsoft started developing a replacement: PowerShell.exe . And functionality-wise it’s wonderful… it can be run interactively, it doesn’t need compilation, it has useful variables, it can access the system’s .NET libraries… it all sounds wonderful… until you try to run the darn thing. Let’s just say: The syntax is frighteningly bad, never mind the documentation plus the fact that for some bizarre reason you’re allowed to run batch files or EXE files, but you need to set an additional policy before you’re allowed to run PowerShell scripts!

 

The C# Compiler

But enough ranting. Thankfully there’s an alternative that’s preinstalled on all modern Windows System: The C# compiler. Yes, it’s there, even if you don’t have VisualStudio installed. Just enter

dir “%WINDIR%\Microsoft.NET\Framework\v4*”

on the commandline and you’ll see the directory of all installed .NET 4 frameworks, each containing CSC.EXE, which is the C# compiler.

Now, you could just use that, but that means a whole lot of temp files since you can’t pipe to CSC.EXE and you can’t run the code immediately. However there’s another way to access it: Through .NET itself via System.CodeDom.Compiler.CodeDomProvider .

 

Using PowerShell to access the C# Compiler

Thankfully, there’s one thing that PowerShell gets right: Giving you access to .NET . It’s not a pleasant experience, but it is possible. And there’s another thing PowerShell gets right: it allows piping anything to it. So we can use a little PowerShell script that invokes CodeDomProvider.CreateProvider to compile our code on the fly and run it immediately.

It’s really pretty simple:

GIST

$opt = New-Object System.CodeDom.Compiler.CompilerParameters;
$opt.GenerateInMemory = $true;
$cr = [System.CodeDom.Compiler.CodeDomProvider]::CreateProvider
   ("CSharp").CompileAssemblyFromSource($opt,
   "public class App { public static void Main() { "+ $input+" } }");
if($cr.CompiledAssembly){
    $obj = $cr.CompiledAssembly.CreateInstance("App");
    $obj.GetType().GetMethod("Main").Invoke($obj, $null);
}else{
    $cr.errors;
}

It’s really very straight forward. Take STDIN, wrap it in a Main function, compile it, run it, report error if there was one during compilation.Through the magic of horrible cmd.exe paramter escaping, this looks a bit differently when passed directly to PowerShell.exe (3 quotes), but you should still be able to recognize it. Just put it in any old batch file (I’m using c#.cmd which I also added to my system’s PATH variable so that I don’t have to enter the whole path each time), but be sure to put it in a single line, because even escaping the linebreak with “^” won’t work for arguments of PowerShell.exe :

GIST

@PowerShell -Command " $opt = New-Object System.CodeDom.Compiler.
 CompilerParameters; $opt.GenerateInMemory = $true; $cr = [System.
 CodeDom.Compiler.CodeDomProvider]::CreateProvider("""CSharp""").
 CompileAssemblyFromSource($opt,"""public class App { public
 static void Main() { """+ $input+""" } }"""); if(
 $cr.CompiledAssembly) {$obj = $cr.CompiledAssembly.
 CreateInstance("""App"""); $obj.GetType().GetMethod("""Main""").
 Invoke($obj, $null);}else{ $cr.errors; } "

Horrible, I know. But it works.

 

Including C# inline in batch files

Now, if you want to actually include any C# in your batch file, it’s surprisingly straight-forward since the cmd.exe ECHO command actually has very straight forward escaping rules. Well, except for | and & , which you best avoid by using .Equals() . But new lines just need to be escaped with a “^” at the end of the line and a space before the final pipe character. OK, that sounds way worse than it actually is:

@echo ^
var a="Hello";^
var b="World";^
var foo=(a+" "+b).ToUpper();^
System.Console.WriteLine(foo);^
if(System.IO.File.Exists("C#.cmd")){^
    System.Console.WriteLine("Hey, you named it C#.cmd too :)");^
}^
 |c#

That’s what a typical call would look like. Again, note the “^” at the end of each line and the space before “|c#”. Remember this and you will be fine. Of course, you can also put the CSharp code in a separate file and use @TYPE to pipe it directly to C#.CMD, so it won’t need any escaping.

 

Issues

Well, there’s obviously the issue of escaping your code if you use ECHO to include it inline, but I really don’t think there’s any way to avoid it.

There are some issues which are mostly due to the C# code running inside the PowerShell process, rather than the CMD.EXE process. Most importantly: You cannot set environment variables without setting them user- or system-wide. You can set the environment variables of the PowerShell process, but these won’t be visible to the parent CMD.EXE process either. Your only way out is to use STDOUT and STDERR and FOR /F to move it to a variable. If that doesn’t work (which may be the case if you want to include the code inline, because escaping inside a CMD.EXE FOR call is incredibly difficult), you’ll need to transport the information using the filesystem.

And since we’re piping the code to PowerShell, STDIN will obviously not be available… so no ReadLine().

 

TODO

Well, obviously support for commandline arguments would be nice at some point, but I haven’t needed it so far.

It would also be nice if the PowerShell could add the class/main wrapper only if there is no method given in the source code. For now I’m simply using two different batch files, c#.cmd and c#full.cmd

 

Hopefully this will make your life a bit easier 🙂

 

JSDoc for Mozilla Firefox Components.interfaces

I freely admit, I’ve been spoiled by VisualStudio and .NET. But right now I need to write some JS code for XULRunner and it’s getting painful:

All the information one needs is available on the Wiki, but I want auto-complete, I want argument descriptions and all the little niceties I’ve come to expect from a development environment. WebStorm does an admirable job at allowing me to document my code in a way that makes all this possible, but it needs JSDoc, not a set of Wiki pages in order to do this.

My solution is less than perfect. I wrote a little parser that tries to scrape the content from the Wiki and transforms it into JSDoc. But since a Wiki is not a structured database, this means interpreting the data. Usually my little parser gets it right, but not always. It’s also a terrible bit of code with lots of little fixes every time I encountered a new style that somebody was using. I’ll release it in time, but right now it’s just too ugly.

However, the result it produced is still apparently the best thing we have available right now, so I’m putting it up here. If there’s any interest, particularly in a permanent solution (which would probably involve keeping the documentation in a standardized format and occasionally syncing it with the Wiki), I’ll be happy to help.

Just add this as a reference to your code in order to use it:

Components.interfaces JSDOC

On the iPad, don’t try to fix scrolling

I have to admit that this really bothers me. Yesterday, I decided to write a little tool to let my boss create his presentations on an iPad by sorting a set of pre-created images. Nothing fancy, but I needed two separately scrolling viewports that are NOT operated with the two-finger-dragging-gesture. Seemed pretty straight forward. Make the elements in the last-touched element position:absolute, so that they scroll with the document, while keeping the rest position:fixed so that they stay were they are. Tried it on various browsers on both Android and Windows and it’s so braid-dead simple that even IE can cope.

The thing that I was worried about was that the iPad would smooth-scroll to the new scroll position that’s needed when you return to a previously-scrolled element (you can’t just move the element to the correct position, because that would usually be negative and that means that you wouldn’t be able to scroll to the left-most parts of the element). Turns out that part worked, but everything else fell to pieces. Switching between fixed and absolute a couple of times with big elements almost always crashes Safari straight away. Plus, after switching a couple of times, the iPad would usually get confused and move the scrollable area to some arbitrary rectangle.

I’m sorry, but I really don’t know how to say this nicely: Apple, get your act together. The iOS browser was great when it came out, but having to worry about scrolling feels like the nineties all over again.

Streaming videos to a MK802 Android system-on-a-stick

I got my MK802 Friday and immediately tried to stream videos… first via SMB shares (works, but the little decoder chip usually isn’t able to keep up with my movies) and then via TravelDevel’s VLC Stream and Convert, which I’m using on my LG P920. Unfortunately VLC S&C is not quiet up to the job of working on a landscape device and the developer has apparently dropped off the face of the earth. So I’ve decided to re-implement it via a tiny web-interface. It’s still very bare-bones, doesn’t work with newer VLC versions, doesn’t manage playlists and is missing any kind of configuration panel (you have to edit the source to change the settings), but it works (at 1024×600@24, 2048kbps). If there’s any interest I’ll release it under GPL and set up a project, but for now you are not allowed to redistribute it; you can only install it on your own machines and only if you accept that I’m not responsible for anything that happens.

Here’s how to get it working:

  • This was only tested on VLC 1.1.0. You can get it here. Install it.
  • Open it and go to Tools/Preferences and choose Show Settings/All at the bottom left.
  • Open Interface/Main intefaces and enable “HTTP remote control interface”. Press Save.
  • Close VLC.
  • Open the directory where you installed VLC and open the http directory.
  • Use your favorite text editor to open .hosts .
  • If you’re using static IPs in your home network, which is highly recommended since it allows you to bookmark VLC’s location on your MK802 and will make your network a LOT more secure, add a new line and write down the IP of your MK802.
  • Otherwise, uncomment 192.168.0.0/16 (remove the “#”). Note that this is very insecure since it will mean that anybody on your network will be able to control VLC which can do a lot of damage to your system. NEVER do it on a public or unencrypted network (in fact, if you’re using an unencrypted network, now may be a good moment to finally enable it).
  • Save .hosts
  • Extract this file to your http directory.
  • Open VLC.
  • On your MK802, open the browser and point it to your PC’s IP (if you don’t know, press Win+R, enter cmd /C “ipconfig & pause” and look for IP-v4 address, which should be something like 192.168.2.64), prefixed with http:// and followed by :8080/vlc.html# .
  • In my case that’s http://192.168.2.222:8080/vlc.html# (Yes, the # is important due to a bug in Android’s URL handler… for some reasons it will never display the hash code, which is what my script uses to keep track of the current directory, unless you enter the “#” manually first; otherwise you won’t be able to bookmark it). Open it and navigate to your desired root folder. Now you can bookmark it. Opening this bookmark will always return you to that folder.
  • Click on the file you want to play or click Play All next to a folder to add it’s whole content to the playlist.
  • Now, you might expect the video to play inside the webpage… sadly that’s not possible due to Android’s broken HTML5/video implementation. In order to see (or return to) the video, you have to press the Video button at the top left.
  • Note that the video will keep on playing on the server even if you leave the page, so don’t forget to press STOP if you value your CPU cycles.

Hope this helps. Thanks to TravelDevel for posting his VLC commandlines, particularly the parts that speed up h264 encoding.

Off-Topic: That’s the way online video should work: Indie Game – The Movie

I don’t really buy any movies online… rather, I buy them on DVD or BluRay and save them to my external hard-drive from there. It’s inconvenient and not how I would like things to work, but it’s the best I can get. Movie studios are paranoid and constantly require newer and stricter copy-protections for online services or they won’t allow them to sell their movies, but disk-based formats are released with a certain copy-protection which is usually quickly broken and cannot be easily upgraded. This way, I can watch my movies when I want to watch them: on the bus with my cellphone (MoboPlayer/Android), streamed via VLC (server) / browser(client) during my lunch break, on my projector/PS3 (PS3 Media Server) with friends or on my net-book (Samba / Mplayer) when I’m in bed. I also get the best quality available with (at least) German and English audio tracks and I can order anywhere. If it’s not available yet in Germany, I can get it from the UK or the US, no problem.

Compare that to the on-line situation. First, I need to find a service that’s available in my country… which means reading about all the better and cheaper services that I can’t get in Germany because, well: I’m in Germany. Then I have to see which one has the movie I want, which usually means not finding one, because the movie/series isn’t released in Germany yet (and may never be). Fast forward 6 to 8 months (if I’m lucky and haven’t forgotten about it and it’s not just available as part of a package). I can find the movie, but there is no audio track besides the German one (which usually is frighteningly bad) and it’s only available in Stereo. If I’m lucky, I can get a HD version in 720p and at a bitrate that’s well below the one for a typical DVD, but usually that “HD” version isn’t available due to licensing issues.

If I should still decide to buy it I get my choice of streaming via either some arcane browser-plugin (Silverlight comes to mind) or a proprietary one which does nothing besides adding security holes to an otherwise secure browser, duplicating its own streaming functionality and making sure that I’m not running a screen-capture program while I’m watching the movie, inevitably using up much of my CPU power and introducing stuttering into the movie. If they feel generous I may get a dedicated client software which does the same thing, but may allow me to buffer more than 3 minutes of the movie so that I don’t have to pause in the middle of the movie.

Of course, if I want to watch it off-line or on anything but a Windows-PC that’s still my problem.

To sum it up: The studios are not making the amount of money from me that they could be making, because they do not offer me the quality of service that would make it easy for me to buy something. I have to order it on DVD/BluRay, wait for it to arrive, copy it and create converted versions for mobile devices. You can imagine that’s not something I do as often as I would, for example, click the buy/download button if it would produce the same result (which is not just a theory, I can actually watch the effect of such an offering by comparing my buyer’s history on Amazon with the one on GOG, which offers DRM-free games). Plus, they have to pay for the disc, shipping, Amazon’s cut and so on and so forth, which all comes out of their margins, so even the little that I do order doesn’t make them as much money as they could be making from a download. It just seems so stupid.

Enter Indie Game – The Movie. This is probably the first time that I bought from somebody who got it right (besides Kookie in the Humble Bundle, but it wouldn’t be fair to count that that, because it was part of a bundle). VHX is handling the distribution and in short, everything is as it should be: I get a streaming version for right away and the most common formats for playback on non-connected devices, with no DRM preventing me from making additional copies in other formats. I bought it as soon as I saw it.

I respect the studio’s desire to protect their work. In fact, I rely on copyright for my work as much as anybody else, but it’s depressing to see an industry self-destruct because of paranoia and a misplaced sense of entitlement. Sure, the people pirating your product are an annoyance and you do not owe them anything, but making your regular customers pay for it is not the way to go if you want to fix it. There are a thousand and one methods that they could employe that would protect them from piracy as much as what they’re doing now (which isn’t working terribly well, as it’s currently easier to pirate than to buy) without alienating their customers. They could add signatures to files (either as metadata or, even better, via Steganography) which would identify the origin of a file if it started showing up on P2P networks, or they could provide a unified DRM as open-source with a free certification program, refusing to license to anybody who wants to sell the movies with an incompatible DRM solution. That would address pretty much any single issue that I have with current DRM systems. And that’s just the stuff on top of my head, but one thing is certain: A licensing jungle combined with proprietary DRM systems that are not compatible between any two services are not the way to go.

I just hope offers like the ones from VHX catch on, so that I don’t continue to get strange looks when I say that I want to pay for my movies and TV series…

Carmageddon

This doesn’t have anything with code or Javascript, but I just love playing the games I played when I was younger. The Settlers, Incubation and Sam and Max are at the top of the list, but in the racing section Carmageddon is just below Screamer and Ignition and even pulls ahead of Twisted Metal 2 and Destruction Derby.

And now they’re making a new game. In case you do not know: Carmageddon is a mixture of car racing and arena fighting, much like the better known Twisted Metal series. It’s also a very tasteless game where you get bonus points for running over pedestrians… you’ve got to realize that this game was made when the discussion about how violence in games influences people in real life was at an all time high. Carmageddon made fun of the whole discussion by making it clear that games have nothing to do with reality: Ironically it was censored in Germany with flying bombs instead of pedestrians because the consors were unable to see that.

Personally I think that the discussion about violence in games was (and still is) heading in entirely the wrong direction: Instead of educating people and teaching them that a computer is not a magic box we still act as if any of that stuff happening on screen was real.
The pixels we see on the screen have no more to do with real persons than children playing cops and robbers with a real gunfight. It’s just polygons, pixels and a bit of math… nothing more. That’s what we should get across and Carmageddon was a not-so-subtle nudge in that direction.

Anyway, they’re still trying to get the funding over at kickstarter.com and while I think the concept of Kickstarter will have to evolve at some point beyond the non-monetary-reward thing they’re doing now, I think in this case there’s enough talent (and they have invested enough of their own money) that it’s likely that they’ll get the game done. They’re not asking for much… $15 for what amounts to preordering the game and I think that’s fair. I’ve given $25 because I want access to the beta, which I still think is very reasonable.

Kickstarter Project Page

Scripting the windows commandline with Spidermonkey made easy

I frequently have to automate really simple tasks, like moving files with a certain filename to another directory and the Spidermonkey shell that now comes with XULRunner (thank you for that Mozilla, building it yourself was time-consuming and annoying) has become an invaluable tool.

Few people know how easy it is to use any Mozilla-JS based program (yes, that includes XULRunner and Firefox) to work with commandline programs. As with any other programming environment you just need to be able to call the popen function of the OS, which runs any command and returns the output as a stream.

Mozilla-JS does not include POPEN. However it does support CTYPES, a system for calling libraries. On Windows POPEN and everything else you need is in MSVCRT.

Opening MSVCRT with CTYPES is easy:

var msvcrt = ctypes.open("msvcrt");

Now you just need to declare what functions you need (_popen, _pclose, feof and fgetc in our case) and what types these require (first parameter is the name, second the interface type, third is the return type and everything else is the type of each argument):

var popen=msvcrt.declare("_popen",ctypes.winapi_abi,
    ctypes.void_t.ptr,ctypes.char.ptr,ctypes.char.ptr);
var pclose=msvcrt.declare("_pclose",ctypes.winapi_abi,
    ctypes.int,ctypes.void_t.ptr);
var feof=msvcrt.declare("feof",ctypes.winapi_abi,
    ctypes.int,ctypes.void_t.ptr);
var fgetc=msvcrt.declare("fgetc",ctypes.winapi_abi,
    ctypes.jschar,ctypes.void_t.ptr);

With this you can very easily build something like C’s SYSTEM call, just with the difference that it will return everything the program outputs through STDOUT:

function system(cmd,raw){
/* Open the program (Windows automatically
   uses cmd.exe for that), use raw mode
   if requested. */
    var file=popen(cmd,"r"+(raw?"b":""));

    var o=""; // STDOUT content
    var end=false; // End of STDOUT reached

/* Loop trying to get a character from STDOUT
   until feof informs us that the end of the
   stream has been reached */
    do{
        var c=fgetc(file);
        if(!(end=feof(file))) 
             o+=c; //Append current char
    }while(!end);

    pclose(file); // Close pipe
    return o; // Return 
}

And that’s really all you need (in this case to call DIR, split by newline and output the first result):

var dirs=system('DIR /B /S');
dirs=dirs.split("\n");

if(dirs.length)
	print(dirs[0].trim());

Zerg Rush Easter Egg

It’s incredibly funny, especially if you know that “Zerg Rush” is a very popular method for rooting Google’s Android OS. I wonder why that connection hasn’t so far popped up in any reports I’ve found about the Google Zerg Rush easter egg

Tip: Relaxing JSON strictness

I’ve decided to start posting about the little things I encounter day by day. Post #1: I needed less strict JSON parsing than Firefox usually does and a little RegEx seems to do the trick for now (it also allows for single word strings à la PHP, e.g. FOO becomes “FOO”:

var findRelevantToken=/(?:"(?:|.*?[^\\](?:\\\\)*)"|([\s,\:\{\[]|^)([a-z\_\$][a-z\_\$0-9]*))/gi;
   str=str.replace(findRelevantToken,function(m,ssep,nqs){
   if(nqs) return ssep+'"'+nqs+'"';
   else return m;
});