Jared Parsons

Warning on lower case type names in C# 11

2022-04-11T00:00:00+00:00

C# 11 is the next version of C# coming in .NET 7, and it is introducing a warning wave that issues a warning when a type is declared with all lower-case letters. This is being done so that in the future the language can begin moving away from conditional keywords and instead use full on keywords instead. The warning is alerting customers to types that may become keywords in future versions.

C# has a strong compatibility guarantee and with that, it has forced us to unnecessarily complicate the language in the past in a few areas. The reason being that whenever we introduce a new keyword into the language it must be a conditional keyword. This means that the C# language must consider both the case when the identifier is used as a keyword and when it’s used as a type

The classic case of this is var. Even though 99.999% of the time var means that it’s an inferred type the language has to consider the case where the user defined a type called var. This is case is on the level of “annoying to compiler developers”. It doesn’t really hamper the language, it just makes compiler developer lives harder.

There are cases that are more impactful though. A frequent request we’ve gotten over the years is async / await is adding unnecessary ceremony to the language. Essentially:

Why do I need async on the method at all? After all I put await x in the body so clearly this is an async method. Let’s remove async and make the language simpler

The problem is that statement isn’t considering the case where the user has class await in their code. At that point await x is a variable declaration, not an operation. That is why async is necesssary, it serves to tell the compiler to ignore any class await in the code and treat awaitas an operator. This is a case where compat has served to make the language more complicated to support obscure scenarios.

When doing record though we hit a bit of a breaking point. This is not a new feature and there is plenty of prior art across the language landscape and that prior art called for a concise declaration syntax. Consider that Java had the following syntax:

record Name(string First, string Last) { }

The compatibility guidelines of C# though forced us to consider the case that record could be a type at which point the above is ambiguous. It could be a type, method signature, lambda, etc. This is a case where there was no out here because it was a hard ambiguity in the language. After a lot of debate, we made the call in C# 9 to say that record was a keyword when used in a type position. That gave us the concise syntax we desired at the expense of breaking users who had class record in their code. This break was conditioned on langversion:9 though so users had to opt into the break. Even so we held our breath a bit when we released it and … no one cared. We made the language simpler, and we lost almost nothing.

There are many other places where we’ve known this will be a problem for future features. Hence rather than take one hit at a time when we release new C# language versions we took the broader view of allowing lower case type names is hampering language evolution. Let’s issue a blanket warning now so users are aware of the potential for these to become keywords in future versions of the language.

That is the motivation for warning here: language simplicity

Exploring borrowed annotations in C#

2019-12-02T00:00:00+00:00

One request I see fairly often for C# is to add the concept of borrowed values. That is values which can be used but not stored beyond the invocation of a particular method. This generally comes up in the context of features which require a form of ownership semantics like stack allocation of classes, using statements, resource management, etc … Borrowing provides a way to safely use owned values without complicated ownership transfer.

This is a feature we explored while working on System C# in the Midori Project in the context of having stack like allocations. The experiment was successful and brought with it significant performance wins for the system. But the experience also taught us quite a bit about the difficulties in introducing ownership concepts into languages and frameworks that didn’t have them designed in from the start.

To help illustrate these difficulties this post is going to focus on what it would look like if borrowing were added to C# for reference types. Borrowing, the concept of use but don’t store, is a necessary pre-requisite for most forms of ownership. Lacking borrowing the more desired features, like stack allocation of classes, wouldn’t be possible.

In this post borrowed references will be denoted with a & following the type name. So Widget is a normal reference while Widget& is a borrowed reference. There is a subtyping relationship between borrowed and normal references meaning a Widget is convertible to a Widget& but not the other way around. This annotation can be applied to locals and parameters. It cannot be applied to fields, return types, or parameters which are out, in or ref.

class Widget { 
    Widget Field;

    void Example(Widget normal, Widget& borrowed) {
        borrowed = normal;  // Okay: converting Widget to Widget&
        borrowed = this;    // Okay: converting Widget to Widget&
        normal = borrowed;  // Error: can't convert Widget& to Widget

        Field = normal;     // Okay: converting Widget to Widget
        Field = this;       // Okay: converting Widget to Widget
        Field = borrowed;   // Error: can't convert Widget& to Widget
    }
}

This simple system enforces that borrowed references have the desired “use but don’t store” semantics. When a value is passed to a borrowed parameter of a method, the caller can be assured that the value is no longer referenced at the completion of the method. That is it cannot be stored into a field, used as a generic argument, returned or smuggled out via a ref / out parameter.

This system is limiting though because there is no way to invoke instance members on borrowed references. The this reference in instance methods is a normal reference. Hence invoking an instance method on a borrowed reference would effectively be converting a borrowed reference to a normal one which breaks the model. To allow for method invocation we’ll let methods mark the this reference as borrowed by adding a & after the method signature. Further any method which overrides or implements a method where this is marked as borrowed must also be marked as borrowed.

abstract class Resource {
    // Method with a borrowed `this`
    public abstract void PrintStatus() &;

    // Normal method
    public abstract void Close();
}

class MyResource : Resource {
    bool Valid;
    public override void PrintStatus() & {
        Console.WriteLine($"Is valid {Valid}");
        MyResource r = this;    // Error: can't convert MyResource& to MyResource
    }

    public override void Close() {
        Valid = false;
    }

    static void Example(MyResoure normal) {
        MyResource& borrowed = normal;

        borrowed.PrintStatus(); // Okay
        normal.PrintStatus();   // Okay

        borrowed.Close();       // Error: can't call a normal method from a borrowed reference
        normal.Close();         // Okay
    }
}

So far this all seems pretty sensible. Borrowed values have the desired “use but don’t store” semantics, have a clean integration into the type system and have a minimal syntax burden.

What happens though when we attempt to leverage this feature in the .NET SDK? Consider as an example string.Format. The parameters to this method are never stored and in practice are often a source of wasteful boxing allocations. This is a classic scenario where borrowing should bring big wins. The parameters can be marked properly as borrowed and then the runtime can safely stack allocate the boxing allocations.

class String {
    public void Format(string format, object& arg) {
        var strArg = arg.ToString();
        FormatHelper(format, stringArg);
    }
}

This example though also reveals a significant problem: the call arg.ToString is illegal because the definition object.ToString is not defined as having a borrowed this parameter. Worse is that the .NET team can’t fix this by going back and marking object.ToString as borrowed. This would be a massive compatibility break because every override of ToString would likewise need to be marked as borrowed.

This compat burden is where borrowing starts to fall down as a feature. It’s not just limited to ToString but virtually the entire surface area of .NET. Borrowed values are significantly hampered because they ….

Can’t call any methods on object GetHashCode, ToString, ReferenceEquals, GetType or Finalize
Can’t call operator==, !=, etc …
Can’t be used as generic arguments. So no List<Widget&>.
Can’t call any method on any interface defined in the .NET SDK surface area.

The non-virtual methods could be fixed by updating their annotation in the framework to be borrowed. The virtual methods and interface definitions though can’t be changed as it would break compatibility. That means object& or any borrowed interface is by themselves is basically useless. They can’t be stored as they’re borrowed and no members can be invoked on them.

This is a pretty significant problem. It means that a good portion of the .NET Framework API parameters can never be marked as borrowed because doing so would make the values unusable. That’s true for object, interfaces or really any unsealed type where virtual methods are used. This means large sections of .NET which are perfect for ownership semantics can never take advantage of them. So much so that it brings up the question of whether this feature is worth doing. Successful uses of borrowing would require significant duplication of the .NET Framework surface area with the only real change being to add borrowing semantics to parameters. Not ideal.

This is the crux of the problem with retrofitting languages with core features like ownership. The problem isn’t just extending a 20 year old language to understand ownership, it’s also about extending a 20 year old SDK. Both present challenges that need to be overcome. In the case of ownership though it’s much more about whether the SDK could adopt it than whether it could be added to the language.

That’s not to say the version of borrowing laid out in this post is complete. It’s in fact lacking a number of features that are necessary for a good borrowing system: relative lifetime annotations, borrowed fields, returning borrowed values, etc … At the same time though those are all relatively solvable compared to the SDK compatibility issues.

string vs. String is not a style debate

2019-04-08T00:00:00+00:00

Often I see developers debating using String vs. string as if it’s a simple style decision. No different than discussing the position of braces, tabs vs. spaces, etc … A meaningless distinction where there is no right answer, just finding a decision everyone can agree on. The debate between String and string though is not a simple style debate, instead it has the potential to radically change the semantics of a program.

The keyword string has concrete meaning in C#. It is the type System.String which exists in the core runtime assembly. The runtime intrinsictly understands this type and provides the capabilities developers expect for strings in .NET. Its presence is so critical to C# that if that type doesn’t exist the compiler will exit before attempting to even parse a line of code. Hence string has a precise, unambiguous meaning in C# code.

The identifier String though has no concrete meaning in C#. It is an identifier that goes through all the name lookup rules as Widget, Student, etc … It could bind to string or it could bind to a type in another assembly entirely whose purposes may be entirely differnt than string. Worse it could be defined in a way such that code like String s = "hello"; continued to compile.

class TricksterString { 
  void Example() {
    String s = "Hello World"; // Okay but probably not what you expect.
  }
}

class String {
  public static implicit operator String(string s) => null;
}

The actual meaning of String will always depend on name resolution. That means it depends on all the source files in the project and all the types defined in all the referenced assemblies. In short it requires quite a bit of context to know what it means.

True that in the vast majority of cases String and string will bind to the same type. But using String still means developers are leaving their program up to interpretation in places where there is only one correct answer. When String does bind to the wrong type it can leave developers debugging for hours, filing bugs on the compiler team and generally wasting time that could’ve been saved by using string.

Another way to visualize the difference is with this sample:

string s1 = 42; // Errors 100% of the time 
String s2 = 42; // Might error, might not, depends on the code

Many will argue that while this is information technically accurate using String is still fine because it’s exceedingly rare that a code base would define a type of this name. Or that when String is defined it’s a sign of a bad code base.

The reality though is quite different. Defining String happens with some regularity as is demonstrated by the following BigQuery:

SELECT  
  sample_path, sample_repo_name
FROM `fh-bigquery.github_extracts.contents_net_cs`
WHERE 
  NOT STRPOS(sample_repo_name, 'coreclr') > 0
  AND NOT STRPOS(sample_repo_name, 'corefx') > 0
  AND NOT STRPOS(sample_repo_name, 'roslyn') > 0
  AND NOT STRPOS(sample_repo_name, 'corert') > 0
  AND NOT STRPOS(sample_repo_name, 'mono') > 0
  AND STRPOS(content, 'class String ') > 0
LIMIT 100

Looking through these results you’ll see that String is defined for a number of completely valid purposes: reflection helpers, serialization libraries, lexers, protocols, etc … For any of these libraries String vs. string has real consequences depending on where the code is used.

So remember when you see the String vs. string debate this is about semantics, not style. Choosing string gives crisp meaning to your code base. Choosing String isn’t wrong but it’s leaving the door open for surprises in the future.

Note: This discussion is not limited to string. It also applies to object, int, long, short, etc … essentially any of the type keywords introduced in C# 1.0.

Is making a struct readonly a breaking change?

2019-03-27T00:00:00+00:00

C# 7.2 added the ability to mark a struct declaration as readonly. This has the effect of guaranteeing that no member of the struct can mutate its contents as it ensures every field is marked as readonly. This guarantee is imporant because it allows the compiler to avoid defensive copies of struct values in cases where the underlying location is considered readonly. For example when invoking members of a struct which is stored in a readonly field.

class Operation { 
    readonly string Name;
    readonly DateTimeOffset Started;

    public override string ToString() => Name + Started.ToString();
}

When calling Started.ToString here the compiler first creates a defensive copy of Started on the stack. The ToString operation is then invoked on that copy. The reason for this is the compiler must assume the worst case which is ToString mutates the contents of the struct and hence violates the readonly contract on the field.

Starting with netcoreapp2.1 though DateTime, and many other types, are now marked as readonly struct. Invocations like the ToString above now occur directly on the field, avoiding the wasteful copy it had before.

These defense copies are small when looked at individually but can quickly add up to a significant performance issue. Particularly in high performance scenarios which make heavy use of readonly and tend to use larger sized struct declarations. Before the readonly struct feature these code bases often had to sacrifice correctness by avoiding readonly to improve perforamnce by avoiding defensive copies. Now though the same code bases can have performance and without sacrificing correctness.

One question that frequently comes up with readonly struct though is whether or not this is a breaking change? The short answer is no. This is a very safe change to make. Adding readonly is not a source breaking change for consumers: it is still recognized by older compilers, it doesn’t cause overload resolution changes, it can be used in other struct types, etc … The only effect it has is that it allows the compiler to elide defensive copies in a number of cases.

That being said there is one scenario to be careful of when applying this feature. One of the requirements is that every field of the type be explicitly marked as readonly. Adding readonly to a field as a part of making the containing type readonly can cause observable behavior changes. When the field type is a non-readonly struct defensive copies will now be made for invocations and this can cause changes to be dropped where previously they were persisted. This has nothing to do with readonly struct but instead is a direct result of making the field readonly.

The CoreFX team ran into exactly this problem when making Nullable<T> into a readonly struct. The T value field was marked as readonly as a part of that process. This turned out to be a breaking change because it meant operations like value.ToString now caused a defensive copy to occur which caused all mutations inside value to be discarded. Eventually this lead to the change being reverted because of the high impact of Nullable<T>.

struct Nullable<T> { 
    readonly T value;
    bool hasValue;

    public override string ToString() {
        // Oops: value.ToString now creates a defensive copy
        return  hasValue ? value.ToString() : "";
    }
}

Again though, this is about marking fields readonly, not the containing type. This type of problem is fairly rare though. Even in code bases where compat is of incredibly high value there have been sweeping changes to mark large blocks of struct declarations as readonly.

The other case where behavior changes can occur has to do with aliasing. This is extremely rare though, only showing up in hypotheticals vs. actual code bases. It is best demonstrated by example:

struct S { 
    static S StaticField = new S(0);
    public static ref readonly S Get() => ref StaticField;

    public readonly int Field;
    public S(int field) {
        Field = field;
    }

    public int M(int value) {
        StaticField = new S(value);
        return Field;
    }

    static void Main() {
        Console.WriteLine(S.Get().M(42));
    }
}

This code will print 0. The invocation of M(42) here occurs on a ref readonly S which means the receiver location is conisdered readonly. This is the ref equivalent of invoking M when the receiver is contained in a static readonly field. The location itself is readonly, the member is not and hence the compiler creates a defensive copy.

When the declaration is changed to readonly struct S the code will print 42. The reason is that there is no longer a defensive copy during the invocation of M. Defensive copies are all about ensuring the target method does not directly mutate the contents of the receiver. But it is still possible for other aliases to the same location to indirectly mutate the contents by assigning into the location.

This is a fairly contrived example though and not one that is likely to occur in many code bases. It is listed here not as a warning against using readonly struct but quite the opposite. It’s meant to demonstrate the level of complication needed to observe the difference.

The take away here is readonly struct is a beneficial annotation, both for performance and correctness, that is very safe to add to your code base.

Profilers and impossible exceptions

2016-11-01T00:00:00+00:00

At first glance some customer crash reports look simply impossible. The code in question is so throughly tested or on such a common code path that it simply can’t be broken. If it were broken customers would be breaking your inbox with crash reports.

Deeper inspection though almost always reveals that indeed it is a bug in the code. Perhaps there is a race condition, a machine configuration issue, a quirk of the OS / BCL the developer was unaware of. Or even more simply, usually in the case of the original author inspecting the code, there is just a straight forward case that was overlooked.

On occasion though no matter how deeply you look at the code the bug report seems genuinely impossible. For example code which throws a NullReferenceException on a path where a null value is impossible. The report indicates a crash but no matter how much you and others look at the code,you just can’t see a way that crash could happen.

Take this particular GitHub issue as an example. The customer presented us with the following call stack:

Unhandled Exception: System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.CodeAnalysis.CompilationOptions.get_Errors()
   at Microsoft.CodeAnalysis.CSharp.CSharpCommandLineParser.Parse(IEnumerable`1 args, String baseDirectory, String sdkDirectory, String additionalReferenceDirectories)
   at Microsoft.CodeAnalysis.CSharp.CSharpCommandLineParser.CommonParse(IEnumerable`1 args, String baseDirectory, String sdkDirectoryOpt, String additionalReferenceDirectories) 
   at Microsoft.CodeAnalysis.CommandLineParser.Parse(IEnumerable`1 args, String baseDirectory, String sdkDirectory, String additionalReferenceDirectories)
   at Microsoft.CodeAnalysis.CommonCompiler..ctor(CommandLineParser parser, String responseFile, String[] args, String clientDirectory, String baseDirectory, String sdkDirectoryOpt, String additionalReferenceDirectories, IAnalyzerAssemblyLoader analyzerLoader) 
   at Microsoft.CodeAnalysis.CSharp.CSharpCompiler..ctor(CSharpCommandLineParser parser, String responseFile, String[] args, String clientDirectory, String baseDirectory, String sdkDirectoryOpt, String additionalReferenceDirectories, IAnalyzerAssemblyLoader analyzerLoader) 
   at Microsoft.CodeAnalysis.CSharp.CommandLine.Csc..ctor(String responseFile, BuildPaths buildPaths, String[] args, IAnalyzerAssemblyLoader analyzerLoader) 
   at Microsoft.CodeAnalysis.CSharp.CommandLine.Csc.Run(String[] args, BuildPaths buildPaths, TextWriter textWriter, IAnalyzerAssemblyLoader analyzerLoader) 
   at Microsoft.CodeAnalysis.CommandLine.DesktopBuildClient.RunLocalCompilation(String[] arguments, BuildPaths buildPaths, TextWriter textWriter) 
   at Microsoft.CodeAnalysis.CommandLine.BuildClient.RunCompilation(IEnumerable`1 originalArguments, BuildPaths buildPaths, TextWriter textWriter) 
   at Microsoft.CodeAnalysis.CommandLine.DesktopBuildClient.Run(IEnumerable`1 arguments, IEnumerable`1 extraArguments, RequestLanguage language, CompileFunc compileFunc, IAnalyzerAssemblyLoader analyzerAssemblyLoader) 
   at Microsoft.CodeAnalysis.CSharp.CommandLine.Program.Main(String[] args, String[] extraArgs) 
   at Microsoft.CodeAnalysis.CSharp.CommandLine.Program.Main(String[] args)
C:\Program Files (x86)\MSBuild\14.0\bin\Microsoft.CSharp.Core.targets(67,5): error MSB6006: "csc.exe" exited with code 255.

A NullReferenceException is occurring inside a call to CompilationOptions.Errors:

public ImmutableArray<Diagnostic> Errors
{
    get { return _lazyErrors.Value; }
}

The only possible source of null here is the field _lazyErrors yet it is readonly and assigned unconditionally in the only constructor of the type. It should never be null. This code in in the initial startup path of csc so there aren’t any whacky multi-threading issues that need to be considered. It’s simple, straight forward code that should never have a NullReferenceException. It’s an impossible bug yet the customer can reliably reproduce the crash.

In cases like this one possibility to consider is the impact of profilers on a .NET process. The CLR APIs allows profilers to modify the IL of method bodies. This is typically done in order to record events such as allocations, method entry, method exit, etc … Profilers though are just code and themselves can have bugs. This includes generating incorrect IL that can open the door to crashes like this.

Checking for the impact of profilers is straight forward when a crash dump is available. For this particular dump you would take the following steps in WinDbg:

!name2ee Microsoft_CodeAnalysis!Microsoft.CodeAnalysis.CompilationOptions.get_Errors

This will dump out a lot of runtime information about the JIT’d method. The value of interest though is the MethodDesc field. In this case it was 000007fe986d9948. This value lets us look at the generated assembly for the method:

!U 000007fe986d9948

This will dump out the following assembly.

000007fe`988f9043 e8989bb1ff      call    Typemock.Interceptors.Profiler.InternalMockManager.getReturn(System.Object, System.String, System.String, System.Object, Boolean, Boolean) (000007fe`98412be0)
000007fe`988f9048 488945d8        mov     qword ptr [rbp-28h],rax
000007fe`988f904c 49b8e833048005000000 mov r8,5800433E8h
000007fe`988f9056 4d8b00          mov     r8,qword ptr [r8]
000007fe`988f9059 48baf02f048005000000 mov rdx,580042FF0h
000007fe`988f9063 488b12          mov     rdx,qword ptr [rdx]
000007fe`988f9066 488b4d10        mov     rcx,qword ptr [rbp+10h]
000007fe`988f906a 4533c9          xor     r9d,r9d
000007fe`988f906d e8469bb1ff      call    Typemock.Interceptors.Profiler.InternalMockManager.ClearRefParameter(System.Object, System.String, System.String, Boolean) (000007fe`98412bb8)
000007fe`988f9072 488b55d8        mov     rdx,qword ptr [rbp-28h]
000007fe`988f9076 488955d0        mov     qword ptr [rbp-30h],rdx
000007fe`988f907a 488b55d8        mov     rdx,qword ptr [rbp-28h]
000007fe`988f907e 4885d2          test    rdx,rdx

It’s not really necessary to fully understand the assembly to diagnose the profiler here. I certainly don’t have the necessary depth with assembly to do so. But one particular item of interest is the call instruction.

The target of the call is the namespace TypeMock.Interceptors.Profiler. That is certainly not a component of the Roslyn code base. Yet it is clearly referenced in the generated assembly. That confirms pretty strongly that a profiler is being used in this scenario to modify the IL in the assembly. Given all the other data it’s pretty reasonable to assume the modified IL is the source of the NullReferenceException.

Cases like this though are pretty rare. I typically investigate around 20 crash reports a month and see this happen less than once a year.

Deterministic builds in Roslyn

2016-04-05T00:00:00+00:00

It seems silly to celebrate features which should have been there from the start. But I can’t help but be excited about adding deterministic build support to the C# and VB compilers. The /deterministic flag causes the compiler to emit the exact same EXE / DLL, byte for byte, when given the same inputs. This is a seemingly minor accomplishment that enables a large number of scenarios around content based caching: build artifact, test results, etc …

Roslyn for example enabled deterministic builds in February. Since then we’ve built a content based test caching system leveraging our deterministic build output that on average provides 72% savings on the time it takes to run tests. That’s a huge productivity boost for developers because our changes can be verified almost 15 full minutes faster now in PRs!!!

Getting a bit ahead of ourselves though. Let’s discuss what exactly the deterministic switch does to the PE output.

Deterministic Compilations

When discussion determinism compilations we need to divide up the contents of the PE into two categories:

The inherently non-deterministic values
Everything else

Everything else includes the order and content of classes, methods, attributes, etc … In the past the compiler has never made any explicit guarantees about the order in which these items are emitted. Yet in practice both Roslyn and the native compiler emitted these values deterministically based on the order in which inputs were received to the compiler. Because this was never guaranteed it was not explicitly tested and hence there were some subtle sources of non-determinism.

Going forward this deterministic ordering is now guaranteed by the compiler irrespective of the /deterministic flag. This does not guarantee a specific ordering such as the first class provided to the compiler will be the first class emitted. Instead it guarantees that given the same source files in the same order the compiler will emit the classes / members in the same sequence in the PE.

That leaves us with the inherently non-determistic values in the PE:

MVID: a GUID identifying the PE which is newly generated for every PE produced by the compiler ¹.
PDB ID: a GUID identifying the PDB matching PDB which is newly generated on every build.
Date / Time stamp: Seconds since the epoch which is calculated on every build.

These three values are the root of non-determinism in the compiler. Everything else has always been, mostly, emitted deterministically. These values though change on every build, even if provided identical inputs, and hence are the root cause of non-determinism in PEs.

There are a small number of indirectly non-determinismic values, such as PrivateImplementationDetails. These derive names from the values above and hence are also non-deterministic. These are secondary issues though, the MVID, PDB ID and Timestamp are the core issues to solve for deterministic builds.

At the core the /deterministic flag simply acts to make these values deterministic while maintaining their original function. In particular the MVID and PDB ID need to still be a unique identifier of the PE and PDB respectively. Using a GUID such as all 0s would be deterministic but would break existing tools which attempted to identify / cache a PE by it’s MVID entry ².

To create the MVID and time stamp with repeatable unique values the compiler uses cryptographic hashes. It takes the content of the PE with the above entries set to 0 and runs it through a SHA1 ³ hash. The resulting 20 bytes are then carved up into a GUID (16 bytes) and a time stamp entry (4 bytes, high bit always set). A similar operation is performed for the PDB ID. This means the above values will be repeatable and unique for a given set of inputs.

The combination of the explicit ordering guarantee and the predictable values for MVID, PDB ID and timestamp allow us to produce fully deterministic PE outputs from the compiler. They will be identical byte for byte.

FAQ

A couple of questions that often come up in this area:

Why not just use all 0s for the timestamp?

This is actually how the original implementation of determinism functioned in the compiler. Unfortunately it turned out there were a lot of tools we used in our internal process that validated the timestamp. They got a bit cranky when the discovered binaries claiming to be written in 1970, over 25 years before .NET was even invented. The practice of validating the time stamp is questionable but given tools were doing it there was a significant back compat risk. Hence we moved to the current computed value and haven’t seen any issues since then.

Why isn’t this behavior enabled by default?

There is one case where the /deterministic can cause a compilation error: the use of * in AssemblyVersionAttribute or AssemblyFileVersionAttribute. The use of * here specifies a value that is required to change on every build (or at least every day). That conflicts directly with /deterministic which pushes the compiler to produce a byte for byte equivalent build given the same inputs. Hence the compiler issues an error notifying the developer of the conflict.

There has been significant discussion around relaxing this case to a warning in C# 7.0 and making deterministic the default. This is on going discussion but I’m optimistic it will happen.

What does the /deterministic switch actually control?

It serves to make the inherently non-determinismic sections of the PE deterministic: MVID, PDB ID and time stamp. All of the other section of the PE are deterministic with or without the switch being present.

Is the PDB deterministic as well?

Windows PDBs still have non-deterministic output. These are emitted by a native component shared by the C++ compiler. Attempts were made to make the output of the PDB deterministic as well but fell short for this release. It’s possible in future releases this will also become fully deterministic.

Portable PDBs are fully deterministic. They were designed with determinism in mind and are fully deterministic in the face of the /deterministic flag.

What if I build from different enlistment paths?

There are a number of cases where the enlistment path of a build will show up in the resulting PE / PDB:

Full path of source files are embedded in the PDB.
[CallerFilePath] embeds full source path as a default argument.
#line directives can include file paths.
The full path of the PDB if generated.

This means identical builds from different enlistment paths will often have different outputs. To fix this the compiler provides the /pathmap: option. It takes arguments in the form of /pathmap:<source directory>=<dest directory>. Multiple pairs can be provided by separating them with a semicolon.

When given this option the compiler will replace the any occurrence of <source path> in file path it writes out with <dest path> instead. This allows builds from different enlistment paths to have identical output.

Note: If producing PDBs then in update 2 the following flag also needs to be provided: /feature:pdb-path-determinism. This is a short term work around that will be replaced with a supported option in update 3.

What options do I provide in MSBuild files?

Here are the MSBuild project file entries for the equivalent command line option:

/deterministic - <Deterministic>true</Deterministic>
/pathmap - <PathMap>source=dest</PathMap>

What inputs to the compiler affect deterministic output?

See the Deterministic Inputs document in the Roslyn repo.

The CLI spec suggests this “should” be newly generated for every PE and practically all tools that output PEs use Guid::NewGuid(). ↩
A common practice in complex build environments. ↩
The use of SHA1 is subject to change in future releases of the compiler. ↩

Successful compilations can have errors

2016-03-24T00:00:00+00:00

Errors are the mechanism by which compilers communicate incorrect program to users. Good error messages educate the user about the issue and ideally tells them how to correct it. This is the optimal situation because it allows users to self correct their code. Bad error messages though typically just state the problem, possibly quite cryptically, without any corrective advice and are little better than the compiler spitting out E_FAIL with a line number.

A frequent piece of feedback from customers is an error message could be improved if the compiler could provide better information about the names and symbols used in the error. For instance:

Implement a better heuristic for guessing the overload the developer intended to use.
Dig through all available namespaces to see if there is type that could match that unresolvable name.
Dig through all available namespaces to see if there is an matching extension method.
etc …

These are valid suggestions but require extra computation by the compiler. In many cases this computation is much more expensive than customers anticipate due to the nature of the language and implementation details of the compiler. When I explain this trade off it often leads to the following exchange:

Them: This error message is terrible and misses the real problem. Why doesn’t it say “do x instead”?
Me: You’re right it absolutely could say that but unfortunately calculating ‘x’ is quite expensive.
Them: That’s ok. This is an error so the compilation is going to fail anyways. I’ll pay an extra second for a better error here.

The problem here is the premise is simply incorrect: encountering an error doesn’t necessarily mean that a compilation will fail. A compilation can actually encounter many errors which are eventually discarded leading to a successful compilation.

The most common reason for this behavior is a side effect of how lambdas and overload resolution interact. Consider the following code, it produces at least one error during compilation but will successfully compile:

void M(Action<int> action) { }
void M(Action<string> action) { }

void Test()
{
    M(x => Console.WriteLine(x.Length));
}

The key here is that lambda expressions don’t have a type by default. The compiler assigns them a type based on how the lambda expression is used ¹. If it is assigned to a local it uses the type of the local, if it’s passed as an argument it uses the type of the parameter, etc … Once the type of a lambda is determined the compiler will then, and only then, evaluate the correctness of the lambda body. For example:

Action<int> del = x => Console.WriteLine(x.Length);

The compiler process this statement in two phases. It will first parse out the lambda expression and verify that it is syntactically correct. There is no evaluation of whether or not x.Length is a legal expression. The parameter has no type at this point hence the compiler simply can’t evaluate it.

The second phase is when the assignment expression between del and the lambda is evaluated. Now the compiler knows the type of the lambda, Action<int>, and it will evaluate the body as if x were typed as int.. In doing so it will issue an error because int doesn’t have a member named Length. Hence Action<int> is not a suitable type for the lambda. To determine this the compiler had to go through the effort of binding the entire lambda body. It’s the only way to verify the type is suitable for the lambda.

Now consider the example above where a lambda is passed to an overloaded method. The compiler now has a pair of parameter types to consider: Action<int> and Action<string>. To determine determine which overload should be used the compiler must consider if the lambda as each of the parameter types:

Action<int>: Error because int has no member Length.
Action<string>: Success as string has a member Length which is a valid argument for Console.WriteLine.

The compiler will choose M(Action<string>) because it’s the only valid overload. Again though the only way the compiler could make this determination was to fully bind the lambda as Action<int> and determine it was invalid due to producing errors. These errors are not just limited to missing members, it can be virtually any error that occurs inside a method body.

This is just one reason why error creation performance is an important issue for the compiler: it occurs even during successful compilations. Even if this weren’t true and errors always represented a failed compilation they would still need to be calculated efficiently. Consider for example environments like Visual Studio that host the compiler. They have a vested interest in displaying errors efficiently.

This is also why C# disallows var x = () => { }, neither the lambda nor the local have a type! ↩

Are private members a part of the API surface?

2016-02-15T00:00:00+00:00

A reference assembly is a slimmed down version of an implementation assembly that contains the API surface but no real code. A program can reference these assemblies at compile time but cannot run against them. Instead at deploy time programs are paired with the original implementation assembly.

Breaking up assemblies into reference and implementation pairs is a useful tool for creating targeted API surfaces. The author is free to exclude otherwise public APIs and hence create a more constrained API surface. This is in fact how targeting PCL, Windows 8, UWP, etc … works.

Another advantage is reference assemblies are significantly smaller than their implementation counterparts. After all they have no code and omit all APIs that are either inaccessible or not intended as part of the API surface. This means SDKs consisting of reference assemblies can be quite small. In order to get them as small as possible developers often ask:

What APIs can omitted such that it won’t effect any code that compiles against the assembly?

On the surface this seems like it has a simple answer: just exclude the types / members that are inaccessible to other programs.

Many developers take this to mean they only need to include public and protected APIs as they are the only ones accessible outside the assembly. This breaks down in the face of InternalsVisibleTo attributes as it makes internal APIs accessible as well. In that case a reference assembly must include the internal APIs to be compatible.

Surely though private APIs can be excluded? There is no PrivatesVisibleTo attribute hence other programs can’t ever access these members. They are nothing more than an implementation detail and can’t effect how other programs compile.

While that is generally true, there is one case where private members have a meaningful effect on other programs: struct fields. The presence, or absence, of these fields can change the way in which the containing struct can be used. The details are often subtle but the scenarios do exist.

Pointer Types

C# permits user defined structs to be pointer types in unsafe code provided the struct meet a specific guideline: it must not contain any fields that are of a reference type. This requirement exists because the .NET GC does not trace through pointers to find object references. Hence references behind pointers would neither be tracked for liveness nor have their addresses rewritten if the object moves during compaction.

Knowing that consider the effect of removing private fields on the following struct:

// Implementation assembly definition
public struct S
{
    private object _field;

    public static S GetValue()
    {
        return new S() { _field = new object() };
    }
}

// Reference assembly definition
public struct S
{
    public static S GetValue()
    {
        throw new NotImplementedException();
    }
}

The implementation assembly definition could never be used as a pointer type (S*) due to the presence of _field. Yet the reference assembly definition meets the C# standard for pointers. This means any program using the reference assembly could legally write the following:

unsafe
{
    S* p = (S*)Marshal.AllocCoTaskMem(sizeof(S*));
    *p = S.GetValue();
}

This is extremely dangerous because now there is a reachable object reference which is untracked by the GC. It’s a crash in the application just waiting to happen at some point in the future!

This behavior difference is also visible in safe code, albeit in a much less severe fashion, as C# allows typeof to target pointer types in any context (safe or unsafe). The following will compile against the reference assembly but not the implementation assembly.

Console.WriteLine(typeof(S*));

Generic Expansion

When constructing generic instantiations that involves structs, C# needs to verify that it doesn’t create a struct layout cycle. For example:

// Assembly 1
public struct Container<T>
{
    private T _field;
}

// Assembly 2
public struct Usage
{
    Container<Usage> Data;
}

The definition of Usage here is illegal because it creates a layout cycle between Container<T> and Usage. Specifically between the fields _field and Data whose types depend on each other cyclically. This makes it impossible to define the layout of Usage and hence C# correctly flags it as illegal.

Now consider what happens if Container<T> is defined in a reference assembly that strips private fields:

// Reference assembly definition
public struct Container<T>
{

}

The lack of the T field means there is no cycle and C# allows the definition of Usage to compile. This would then result in an error at runtime as the CLR can’t represent this definition ¹.

This particular scenario is interesting because it’s not limited to C#. It affects any .NET language that implements generics: C#, F#, VB and C++/CLI.

Definite Assignment

Definite assignment is the process by which C# ensures programs only use variables which are properly initialized. The spec rules for definite assignment can be a bit complicated. In practice though they can be summarized as variables must be assigned or used in an out position before they are referenced as a value.

Structs are an interesting case because definite analysis only requires that all of the fields of a struct are assigned a value ². This can be done by calling a constructor, assigning default(T) or initializing all fields by hand. This last case is interesting because it allows for structs to be considered initialized without every being assigned a value as a whole:

struct Point
{
    public int X;
    public int Y;
    public override string ToString() => $"{X} - {Y}";
}

Point p;
Console.WriteLine(p); // Error! p is not initialized
p.X = 0;
p.Y = 0;
Console.WriteLine(p); // Okay, it's initialized at this point

This logic also applies to struct values which have no fields. Instances of such types are trivially considered to be initialized:

struct Example
{

}

Example e;
Console.WriteLine(e); // Okay, e is trivially initialized.

Knowing that consider what happen to the usage of a struct consisting of only private fields if they are stripped away:

// Implementation assembly definition
public struct Rectangle
{
    private int Width;
    private int Height;
}

// Reference assembly definition
public struct Rectangle
{

}

To other programs Rectangle now appears as an empty struct. That means it can be used without any assignment (as Example was above). This creates an observable difference between programs that compile against the reference and implementation assembly. Code which is legal in one context is illegal in the other.

To be fair here: this is not a violation of IL but only C# rules. Other languages, notably VB, specifically allow for using variables before they are properly initialized. It does however cause observable breaks in C# applications.

Conclusion

The scenarios above are just the most common, and severe, consequences of stripping private fields from structs in reference assemblies. There are several other less severe ones that are affected:

Explicit struct layouts: can’t be done if there is a reference type field.
Interop: developers needs to know true size to design interop correctly.

All of these scenarios though add up to one unfortunate conclusion. Private structs fields cannot be considered just an implementation detail. They are instead an observable part of the struct’s contract.

Experimentally this throws a TypeLoadException ↩
This was done in part to ease the porting from C programs. ↩

Observing a null this value

2015-03-11T00:00:00+00:00

One of my favorite bits of .NET trivia is whether or not it is possible to observe a null value for this? Most developers I ask either say no, or yes but it requires incorrect IL / unsafe code. Since I’m writing this post you can probably guess that the answer is actually yes, this can indeed be null.

To demonstrate this behavior let’s start with a simple command line application:

class Program
{
    void Test()
    {
        if (this == null)
            Console.WriteLine("this is null");
    }

    static void Main(string[] args)
    {
        Program p = null;
        p.Test();
    }
}

No tricks here. Compiling and running this program will result in a NullReferenceException on the p.Test call exactly as a developer would expect. But how is this exception being generated? Lets take a look at the IL for this particular call using ildasm.

IL_0003:  ldloc.0
IL_0004:  callvirt   instance void NullThis.Program::Test()

The Test method is being invoked via the callvirt instruction. This performs a virtual dispatch of the target method. Part of the contract for this instructions is to throw a NullReferenceException if the target object, in this case p, is null. Hence this is the source of the exception.

But is callvirt necessary here? This is a non-virtual method so surely the virtual dispatch is unnecessary overhead. Let’s fix that by changing the instruction to call.

IL_0003:  ldloc.0
IL_0004:  call   instance void NullThis.Program::Test()

The new IL is legal and PEVerify clean. But what happens if we pass this through ilasm and run the resulting program?

$>ilasm /out:NullThis.exe /quiet NullThis.il 
$>NullThis.exe
this is null

Viola, our program has observed a null value for this.

The reason for the behavior change is the call instruction simply does not check for null as does callvirt. It passes along the target object as this without any inspection. This is one of the reasons why languages like C# will emit a callvirt instruction even for non-virtual methods. It serves as a cheap way of doing a null check on the target object.

Now after reading this some developers might wonder if they should start adding null checks for this to their code? No, please don’t. The standard .Net compilers (C#, VB, F#, etc …) will not emit IL that can observe a null value for this. Observing this behavior requires either hand crafting IL, using C++/CLI tricks or tricks with PInvoke and unsafe code. Not a situation that the majority of .NET applications out there will encounter.

The curious case of Task.Run

2015-02-23T00:00:00+00:00

Recently I was confused about the interaction between Task and CancellationToken. In particular I couldn’t remember if a Task which was already running was marked cancelled as soon as the associated CancellationToken was cancelled or if it waited until the Task completed. The documentation wasn’t much help so I decided to write up a quick program to test out the behavior:

static void Main(string[] args)
{
    CancellationTokenSource cts = new CancellationTokenSource();
    Task t = Task.Run(() => { while (true); }, cts.Token);
    cts.Cancel();
    Console.WriteLine(t.Status);
}

This Task never completes so it should answer my question pretty easily. It will either print out “Canceled” (does immediately cancel) or “WaitingToRun / Running” (waits for completion). To my surprise though this printed out:

WaitingForActivation

Say what? A Task created with Task.Run should be started automatically, it should never be waiting for activation. That’s one of the advantages to this API.

This behavior had me puzzled quite a bit. Enough so that I ended up emailing Stephen Toub about it. Between the two of us we were able to track the behavior down to a quick of both the C# compiler and the Task APIs.

The C# quirk involves how the lambda conversion is processed. In this case the compiler detects the lambda never returns because it has an infinite loop. The compiler allows lambdas that never return to convert to delegate of any return type that is otherwise compatible ¹. Hence it can convert equally well to Func<Task> as it can to Action.

This comes into play when we consider all of the overloads available for Task.Run:

public static Task Run(Action action);
public static Task Run(Func<Task> function);
public static Task Run(Action action, CancellationToken cancellationToken);
public static Task Run(Func<Task> function, CancellationToken cancellationToken);
public static Task<TResult> Run<TResult>(Func<Task<TResult>> function);
public static Task<TResult> Run<TResult>(Func<TResult> function);
public static Task<TResult> Run<TResult>(Func<Task<TResult>> function, CancellationToken cancellationToken);
public static Task<TResult> Run<TResult>(Func<TResult> function, CancellationToken cancellationToken);

The generic ones will be eliminated because the compiler can’t infer a type for them. The ones without a CancellationToken parameter will also be eliminated because they don’t match the argument count. That leaves the compiler choosing between.

public static Task Run(Action action, CancellationToken cancellationToken);
public static Task Run(Func<Task> function, CancellationToken cancellationToken);

The lambda due to its infinite loop can convert to each delegate type. The compiler considers the conversion to Func<Task> better though because it has a return type and Action does not ². Hence this overload is picked.

Now if we look closely at this API the return type is a little off. Most of the other overloads that take a Func<X> delegate have a return type of Task<X>. The Func<Task> overload though just returns Task.

It isn’t doing this by taking advantage of the inheritance relationship between Task<T> and Task but instead it’s calling Unwrap under the hood. This creates a proxy Task which presents the original Task<Task> as single item.

So the quirk of the Task API is that there are two Task values here, not one. The first is the Task<Task> which is created to run the lambda is indeed “WaitingToRun / Running”. The second is the Task which is dependent upon the first and hence is “WaitingForActivation”.

To fix both of these I just needed to change the Task.Run call to pick the correct overload:

Task t = Task.Run((Action)(() => { while(true); }), cts.Token);

Now I get the answer I originally wanted: WaitingToRun. Cancelation is not prompt in this case.

Isn’t overload resolution grand?

One other case being method bodies that unconditionally throw an exception. ↩
Documented in section 7.4.3.3 of 3.5 language spec. ↩

Determing MSBuild property values

2015-01-18T00:00:00+00:00

Debugging MSBuild is usually a three step process:

Turn on diagnostic verbosity (/v:diag).
Piping the gargantuan amount of MSBuild output to a file.
Using your favorite text searching tool to find that one line necessary to diagnose the problem.

This method is effective but laborous. However this method really only works if the build file can get to the point of processing <Target> elements. If MSBuild fails before that even /v:diag will fail to output much, if any, information.

When MSBuild fails that early it’s typically because of malformed elements such as an empty Project attribute on <Import> element. Happens most often when the project file isn’t accounting for all the possible values of the input MSBuild properties. But with /v:diag procuding no helpful output it’s hard to determine what the actual value of the troublesome properties are.

In that situation the following really simple trick can be used to print out the property values. Simple replace the contents of the build file in question with the following:

<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="4.0" DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <Target Name="Build">
    <Message Text="VisualStudioVersion value is '$(VisualStudioVersion)'" />
  </Target>
</Project>

This takes the malformed elements out of the equation and sets the Build target to just print out the values in question. It’s printf style debugging but it’s simple and effective.

Looking forward to a better StyleCop

2015-01-14T00:00:00+00:00

A consistent coding style is one of the most undervalued components of a maintainable code base. Code should always be optimized for readability as developers spend far more time reading code than writing it. Having a consistent style helps here because it establishes conventions and locations for well known programming elements.

When taken individually items like the naming of fields, the position of braces, the casing of names, etc … are all small neglible issues. Any time spent debating which is right or wrong is time that could be better spent doing virtually anything else. Yet when these items are taken as a whole they serve to create a dialect within the language.

Just as it’s hard to understand someone speaking in a dialect that differs from your own, it’s hard to read code that significantly differs from your dialect. Or at the very least, it slows down the reading process. You’re looking in the wrong for the wrong name pattern, searching for where fields are defined, looking for the wrong naming pattern, etc … The structure of the code is constantly distracting from what it is actually doing.

This gets even harder as the number of styles within a code base increases. Juggling two styles is annoying but once you’re juggling five it’s a real hamper to productivity. Code bases with no enforcement end up with as many styles as there are developers. It only takes a few years before that number gets into the teens and the code is as messy as a college dorm room.

Coding styles help prevent this by adding a level of consistency to a project. It makes the transition between files as smooth as possible because it lets deveolpers focus on what the code is saying, not how or where it is saying it.

A coding style though is worthless unless it is ruthlessly enforced. Developer diligence is simply not a good method of enforcement here. Coding styles can be highly contenious issues and having to constantly square off over them on code reviews and with new hires is a mentally draining activity. Over time even the diligant developers wear down and the code once again begins to devolve into individual preferences.

This is why I’m an unabashed lover of StyleCop. It is simply the best tool around for defining and enforcing style guidelines in a code base. It can be run on the command line, in the IDE and as a part of a checkin verification system. This removes the need for developers to constantly square off over the issue. Get the style right or your change simply won’t pass the necessary tests.

As much as I like StyleCop though I’m enormously frustrated by it when I have a build fail because:

Didn’t add a space between an if and a (.
Had an extra newline at the end of the file.
Put a brace on the wrong line.
Namespaces weren’t sorted correctly.

All I wanted to do was F5 my latest change and see if it fixed the bug I was working on. But before I can test out a functional change to the source tree I need to spend a few precious minutes making style edits to my code. Why is this silly style issue blocking me from testing my change???

I find annoynances like this are often the root of opposition to adding style enforcement to a code base. Developers are fine, even if it’s reluctantly, with having a coding style. What they dread is constantly fighting the tools to get their job done.

And they are right to feel that way. Fixing style issues is a tedious, repetitive task. Changing the code to be in conformance to the style is needed is easily automatable. We should be relying on tooling here, not manually intervention by annoyed developers.

Many of the fixes to style violations don’t require complex solution wide analysis; just a parser and code generation API ¹. These changes amount to little more than code formatting. The IDE is constantly formatting my code as type, it should be fixing my style violations at the same time.

What I don’t want is to see fixes move into smart tips. Initiating that type of fix still requires manual intervention on my part. Just fix the code without even bothering me:

Save a file: fix the style violations as a part of the save.
Close a method brace: fix the style violations just like you’d format the method.

Not all style fixes are simple enough to be done implicitly. Those more explicit ones could be done in batch via an explicit IDE command. Or from a command line tool that runs over a solution.

This is where I believe style enforcement tools need to go in the future. They absolutely must keep their checkin enforcement capabilities. But they should also evolve so style conformance is as automated as possible and those checkin tests are rarely, if ever, hit during normal development.

Can you say Roslyn? ↩

A new, yet familiar, adventure

2015-01-09T00:00:00+00:00

For the last four years I’ve been a part of a research team exploring extensions to the C# language. The effort focussed on adding features that improved the reliability, performance and overall correctness of the language. Like any good research project we had some features that were incredibly successful, some which were so-so and others that … well … they failed spectacularly!

The project was amazing to work on but like all good things eventually it must come to an end. Research is great fun but a part of me always missed being a part of a shipping product: engaging with customers, the joys (and horrors) of shipping code, etc …

As such I recently decided to accept a position back on C# Language team. Besides normal language work a part of my job will be trying to integrate some of the goodness that came out of our research back into the language.

I do want to emphasize trying above. Certainly not all of our work, or even most of it, will make it into the shipping language. There were plenty of successful features specific to our projects that just don’t make sense in the general context of C#. Even the features I hope to bring back will inevitably need to change a bit to be ship ready.

For the moment though I’m heads down with the rest of the team trying to get Roslyn and Visual Studio 2015 out the door. In the near future the design for C# 7.0 will begin and I will begin working on prototyping these features.

That does raise the question though: what were we working on in research?

With the exception of the OOPSLA paper I’ve been mostly silent since I joined the research team. The need to be quiet about your work is the big downside to research (especially for chatty people like me). But now the research is concluded and I’ll be able to once again blog about my work including most of what I did over the last four years.

Even though I’m incredibly excited to be rejoining the Managed Languages team I will sorely miss my old team. My language team was a part of a larger operating system research effort that had shared goals around reliability, performance and correct by construction. The overall team had a level of rigour, drive and dedication to quality that I’ve never experienced before (but desparately hope to experience again). I will definitely miss the work and the people involved in it.

Floating point casts are rarely redundant

2014-12-22T00:00:00+00:00

Good developers are always on the look out for unnecessary code. Both to avoid in their own code and to help others avoid during code reviews. One such example is redundant casts: a cast where the type of the expression and cast are the same type. For example:

float x = GetValue();
float y = z / (float)(x * 2);

The result of x * 2 is a float. Hence the (float) cast here is not changing the type, it is seemingly just adding extra words to the code. Given the simplicity of the expression and context it’s hard to argue that it’s making the code more readable. It’s just a cast that adds no value to the code and other developers may call for it to be removed.

But does it really have no value? Let’s take a look at the IL generated by the above code:

IL_0008: ldc.r4 2
IL_000d: ldloc.0
IL_000e: ldc.r4 2
IL_0013: mul
IL_0014: conv.r4
IL_0015: div
IL_0016: stloc.1

Now take a look at the IL after that cast is removed:

IL_0008: ldc.r4 2
IL_000d: ldloc.0
IL_000e: ldc.r4 2
IL_0013: mul
IL_0014: div
IL_0015: stloc.1

In the second set of IL instructions the conv.r4 instruction is missing. This instruction is used to convert the top of the evaluation stack to a float. In this case the top of the stack is already a float so is this instruction, like the cast used to produce it, redundant?

It turns out that this instruction is not redundant, it’s actually significant to the value of the expression.

The CLI specification in section 12.1.3 dictates an exact precision for floating point numbers, float and double, when used in storage locations. However it allows for the precision to be exceeded when floating point numbers are used in other locations like the execution stack, arguments return values, etc … What precision is used is left to the runtime and underlying hardware. This extra precision can lead to subtle differences in floating point evaluations between different machines or runtimes ¹.

This is where the extra conv.r4 and conv.r8 instructions come in. Typically they are used to coerce non-floating point values into floating point values. One of their side effects though is the resulting value will have the exact precision specified by the type. This means when applied to a floating point value on the evaluation stack it will truncate it to the specified precision.

Hence the extra float cast in the original code is also not redundant, it is instead ensuring that the result of the multiplication operation is exactly 32 bits. A double cast can be used in exactly the same fashion. This is typically unneeded but can be vital in floating point intensive applications.

This brings up another question: Is this behavior guaranteed by the C# specification? At this time the answer is no, it is simply an implementation detail. If it was guaranteed I think the most logical place would be in section 4.1.6. I don’t think developers need to spend much time worrying about this. Both the original and Roslyn based C# compiler exhibit this behavior and changing it would be inviting a hard to track down back compat break.

Aside: I actually discovered this behavior because I inadventently broke this with a change to the 5.0 compiler. One of the test suites, which had nothing to do with floating point, showed an IL diff where several conv.r4 instructions were removed. At a glance they seemed unnecessary, the code I had changed had no comments about the behavior, the C# specification didn’t mention it and the Roslyn compiler also did not emit them.

In order to be thorough I emailed the Roslyn team to ensure the change was deliberate and after some discussion it turned out it was not. Eventually the decision was to keep the emitted conv.r4 and potentially update the specification to codify the behavior.

Bugs resulting from these differences can be quite madenning to track down. ↩

Use strong name references in a VSIX project

2014-11-17T00:00:00+00:00

If you create a VSIX project using the most recent version of Visual Studio on your machine it will spit out a reference section in your project file containing the following:

<Reference Include="Microsoft.VisualStudio.CoreUtility" />
<Reference Include="Microsoft.VisualStudio.Text.Data" />
<Reference Include="Microsoft.VisualStudio.Text.Logic" />
<Reference Include="Microsoft.VisualStudio.Text.UI" />
<Reference Include="Microsoft.VisualStudio.Text.UI.Wpf" />

This inocuous looking section can be the source of major debugging pain down the road. It is one of the first items you should change when creating a VSIX project.

This section is referencing VS SDK DLLS in a non-strong name fashion. This means any DLL with a matching assembly name can satsify the reference independent of what version it may be. The 2010 VS SDK assemblies can satisfy it as easily as the 2015 VS SDK assemblies. Which DLL gets chosen as the reference here is purely up to MSBuild + VS SDK targets files ¹ and they prefer newer Visual Studio SDKs over older ones. For example on a machine with Visual Studio 2013 and 2015 MSBuild will always prefer the 2015 SDK even if you are developing in 2013.

In general this isn’t a problem because developers tend to use the most recent Visual Studio instance on their machine for active development hence the versions line up (by pure coincedence). Developers eventualy get into trouble though when one of the following happens:

They install a new Visual Studio version, say Visual Studio 2015, on their machines
The original developer, or another person wanting to contribute, clones the project onto a machine with a newer version of Visual Studio
Project starts running on a CI system like AppVeyor which has every version of Visual Studio installed

When any of these happen the developer will F5 their VSIX project using the older version of Visual Studio and be greeted with a MEF composition error. Nothing has changed in the source code of their project but suddenly it no longer works. Even stranger is that the project will continue to work on other machines. It’s a highly frustrating situation that usually leads to several rounds of MEF debugging.

What is happening here is that MSBuild is satisfying the assembly references with the newest VS SDK. They have a higher priority and because the reference is using a weak name MSBuild will pick it as the winner. These assemblies are all backwards compatible so the build succeeds. The F5 operation though will still launch Visual Studio 2013 and the 2013 editor assemblies. Eventually this will lead to a type load error because your VSIX is bound to 2015 assemblies and there is no binding redirect in place.

The fix for this problem is very simple: use a strong name when defining VS SDK references.

<Reference Include="Microsoft.VisualStudio.CoreUtility, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a, processorArchitecture=MSIL" />
<Reference Include="Microsoft.VisualStudio.Text.Data, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a, processorArchitecture=MSIL" />
<Reference Include="Microsoft.VisualStudio.Text.Logic, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a, processorArchitecture=MSIL" />
<Reference Include="Microsoft.VisualStudio.Text.UI, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a, processorArchitecture=MSIL" />
<Reference Include="Microsoft.VisualStudio.Text.UI.Wpf, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a, processorArchitecture=MSIL" />

This reference can never be ambiguously matched: it will either bind to the 2010 VS SDK assemblies or MSBuild will issue an error. Making this change will save you many headaches and hours of MEF debugging on your VSIX project.

To keep it simple I’m going to refer to the combination as MSBuild in the rest of the post ↩

Experiences using F# in VsVim

2014-10-31T00:00:00+00:00

A few days ago while discussing VsVim with a coworker it occured to me that I’d been working on this project for 5 years now. It was a bit startling for me because it feels like just yesterday when I finally got permission to release it to the public. After reflecting for a few minutes I decided that I wanted to write a couple of posts on this project and how it shaped up over time.

What better place to start than how F# impacted this project?

Why did I choose F# for VsVim?

I started working on VsVim at the end of the Visual Studio 2010 release cycles. That was easily the toughest release cycle I ever went through and towards the end I was approaching a major burn out. I needed a fresh new side project to give me some release and boost my morale¹.

When I took a step back to look for a project to start on there were a couple of brand new items in Visual Studio that interested me:

WPF editor and MEF extension model
A functional .Net language named F#

Both of these were shiny new toys that looked like they needed to be experimented with. Why not just use both in the same project?

F# in particular intrigued me. The language team sat just down the hall from me and through the 2010 release cycle I’d gotten to know them pretty well. They were all smart, engaging developers who were eager to chat about what they were doing. The language itself looked very appealing and the fact that I had 0 practical experience with functional programming seemed like a problem worth fixing.

I stared by writing a couple of toy programs in F# and this only increased my interest in the language. Very quickly I settled on the idea of writing a small vim emulator extension in F#.

Embracing the language

As I stated before I had 0 practical, or really any, experience using a functional language. My background was a heavy dose of C/C++ and Java with a side helping of scripting. F# was a brand new world for me and I was determined to dive in full force.

That pretty quickly backfired on me. This was before Visual Studio 2010 was released so F# documentation was still very thin and there were very few samples to draw from. I was spending all of my time trying to translate complex C# COM interop and WPF into F# using nothing but the language spec. It was painfully slow and eventually forced me to divide up the project:

F# for the core vim logic
C# for WPF and VS shell integration

This break down let me to focus on doing algorithms in F# and all the WPF / interop code in C# where I could mostly copy existing patterns. Now I could focus on actually learning F#.

Even with the reduced scope progress was still slow. The syntax was strange, well known types had funny names and were in the wrong place, signature files were odd and the heavy use of type inference and _ values made the few samples that did exist difficult to follow. My reliable guide through this process was the F# spec and Brian McNamara’s blog whose samples I tried to emulate in style and approach. Overall though it was very rough sledding.

It took a couple weeks before I could code for a solid ten minutes without jumping back to the F# spec or email for help. After one month it started getting better and I was spending less time fighting the syntax and more time working on algorithms. The progress was slow but steady.

The real change came about 3 months in when F# finally clicked for me. Suddenly I wasn’t translating F# anymore I was thinking in it. That’s when my productivity started to soar and I really began to leverage F# to make my code better:

Find a place where the editor API sometimes return null? Wrap the return in an option type.
Tons of nested match blocks? Simplify with a maybe monad.
Function too complex? Leverage lambdas as nested functions.
Types too intertwined? Break them apart with signature files.
Null reference exceptions? None of that going on here.
Immutable values whenever possible.
Discriminated unions for so many scenarios.

Discriminated unions in particular were a mental revolution for me. Enums with type safe state is a feature I’d long craved in other languages but never had a name to attach to it. I’ve since found countless uses for them in F# and miss them terribly when I’m in any other language. If I could add one and only one feature from F# back to C# this would be it.

Once I started thinking in F# there was really no turning back. It was still about six months in before my F# productivity matched that of C# but the flexbilitiy and expressiveness of F# more than made up for the gap.

Open source contributions

The one area where F# did not shine for me was OSS contributions. The desire to contribute to VsVim was there but developers often saw F# as a barrier to contribution. Several times a month I would get emails along the lines of:

Man I’d love to contribute by fixing that bug but I just don’t know anything about F#

This may seem strange now given F#’s rise in popularity but at the time it was a brand new language. The documentation was still weak and there weren’t a lot of real world sample code bases to draw from ². The language was a barrier developers didn’t want to cross.

The problem was severe enough that ~2.5 years into the project I had begun to outline a plan to convert all of my F# to C# over the course of a couple releases. I was very happy with the language but the prospect of being the sole significant contributor to the project over the long term wasn’t something I was looking forward to.

But then the tide started turning and doing so relatively fast. As the community around F# grew up the documentation got better, F# talks started showing up at conferences, more OSS projects came into existance, great blog series were written and then suddenly developers were sending me patches.

The really interesting part is that most of the developers still hadn’t written F#. But now they were willing to take the plunge and learn enough of the language to get their fixes submitted. I can only presume this is because the language simply appeared more mainstream and developers felt like it was a valuable use of their time to learn the language. Whatever the reason though I couldn’t be happier.

I think F# is still attaches a higher bar to contribution than C# does, but that bar is significantly lower now and getting lower every day.

Looking back

When I look back on my experiences with F# I’m amazed at how much learning it changed me as a developer. I already had a strong focus on correct, testable, maintainable code with a strong focus on simple, maintainable types / functions. F# taught me to simplify even further by seeing types as data instead of a collection of behaviors. Sure there is a place for mutating objects but I now default to seeing them as data first and behavrios only when it really makes sense in the greater context of the program.

It also showed me how a type system can be used, and sometimes abused, to create more robust APIs. Unfortunately concepts like non-null, discriminated unions and the like don’t map well in C#. But it is possible to take smaller steps like embracing call backs, using default values instead of null, leveraging immutable types etc …

I think the best way to summarize how I feel about F# is that it’s changed the way I approach coding in other languages. This is true from C# to C++. I couldn’t be happier that I decided to write VsVim in F#. If you’re starting a .Net project soon and have never used a functional language I highly suggest you give it a try as well.

Because when you’re burnt out coding another coding project is a great way to releax ;) ↩
For a long time VsVim was one of the largest OSS F# projects ↩

Invalid Uses Of a Type

2014-10-15T00:00:00+00:00

I joined a discussion recently about a new API being added to a type with well established semantics. The API seemed to violate the key invariants of the type and I was curious about the justifications. Of the various reasons given one in particular stood out to me as questionable:

The type can already be used in this manner hence this API adds nothing new to the equation, it just standardizes the behavior

Typically this is a really good justification for adding an API. Centralizing similar code snippets into a single API lowers maintenence cost, reduces code duplication, etc … As previously stated though this seemed like a violation of established semantics so I dug into the provided samples. They all fell into the following categories:

Unsafe Code
Unverifiable Code
Private Reflection

This is an argument I’ve heard before but it’s not one that adds any weight for me. Code samples of this nature are intentionally violating the type system and should not be considered a valid use of the type.

If samples of this nature were considered valid then they could be used to justify practically any API on a type. For example this argument could be applied to adding a setter on the indexer of string. After all with unsafe code I can already mutate its contents so by this reasoning mutations are valid:

string s = "cat";
unsafe {
    fixed (char* i = s) {
        // "cat" becomes "bat"! 
        *i = 'b';
    }
}

Clearly though string is an immutable type and making it mutable would invalidate loads of existing code in the wild. No developer out there designed their programs considering the impacts of a string being mutated. That would be a waste of brain power.

That is why I consider this type of example to be flawed. The above listed techniques exist purely to violate the verifiable type system and to bypass object accessibility. By definition they are using a type in a way that it was never meant to be used. These are simply not valid samples.

Easy Motion for Visual Studio

2014-09-23T00:00:00+00:00

Easy motion is a plugin for Sublime that allows for quick and simple keyboard navigation within a file ¹. Just 3 key strokes can take you to any visible letter. No neeed for complex regexes or patterns, all you need to know is the letter that you want to navigate to.

A user alerted me to this sublime plugin a month or so ago and I immediately started using it. I absolutely loath touching the mouse or arrow keys during development because I feel it slows me down. Typically I use vim style editors to avoid the mouse. But the steep learning curve of vim shouldn’t be a prerequisite for efficient keyboard navigation. Easy motion represents a great middle ground here for developers who want to get more productivite with a minimal learning curve.

So a few weekends ago I sat down and coded up an Easy Motion clone for Visual Studio. Been toying with it for a few weeks now and it’s ready to be shared out more generally.

Here is a simple example of how to use EasyMotion to jump around within a file. Let’s start with a "Hello World" program that has the cursor positioned at the end of a using directive.

Now I want to move the caret to the start of Console.WriteLine. Instead of moving my hands to the arrow keys, or even worse grabbing the mouse, I initiate an easy motion search by pressing Shift+Control+;.

In response the editor added a status line asking me for the character I want to search for. I type C as it is the first letter in Console.WriteLine.

There are many occurences of C in the file and Easy Motion distinguishes between them by overlaying every occurence with a new letter (a-z). To jump to a specific instance of ‘C’ I simply type in the letter which overlays the ‘C’ I want to jump to. In this case it is ‘I’

Now the caret is positioned exactly where I wanted at the start of Console.WriteLine. No arrow keys, no mouse, just 3 quick keyboard touches and I’m there

The Sublime plugin is itself a clone of a vim extension ↩

Releasing VsixUtil

2014-09-19T00:00:00+00:00

A good portion of my free time, and not so free time, is devoted to Visual Studio extensions. In addition to actively developing them I’m always dogfooding my extensions and those of other developers. On any given day I probably update Visual Studio 3-5 times on various machines with new builds, proposed bug fixes, forks, etc …

As with any other repetitive task I like to script this as much as possible. Building, uninstalling and reinstalling extensions is tedious work. Much better to just write a powershell script once to do it and just keep running that.

Unfortunately there is really no good way to script the installation of a VSIX. Scripting requires command line tools and there just isn’t a good one for VSIX. Visual Studio does come with vsixInstaller but its just not suitable for my scenario

Each build is tied to a version of Visual Studio. This means it can’t be xcopy deployed to a machine with an arbitrary version of Visual Studio and be expected to work.
At the core it is a GUI application, it just happens to be usable from the command line. As such it’s functionaly asynchronous and can’t be used for error detection.
It doesn’t support deployment to alternate registry hives

The first issue is a real problem for me. I’m not sure I have two machines anywhere with the same combination of Visual Studio installs on them. Adding VS detection logic to every sinlge script I write to find the correct vsixInstaller isn’t a maintainable solution. I really need a tool that will just work no matter what version(s) of Visual Studio are on the machine.

This lead me to develop a simple command line application: VsixUtil. This is an xcopy tool that will function on a machine with any version of Visual Studio installed (even Dev14). The command line is straight forward

Install: vsixutil [/rootSuffix name] /install vsixFilePath
Uninstall: vsixutil [/rootSuffix name] /uninstall identifier
List installed: vsixutil [/rootSuffix name] /list [filter]

A version of this tool has been the heart of my VsVim dogfooding scripts for some time now. Recently I decided to make it more general use to support my increased dogfooding habbits and now it is ready to be shared with others.

Happy Dogfooding!

Source: https://github.com/jaredpar/VsixUtil
NuGet: https://www.nuget.org/packages/VsixUtil

Vim key maps don't have comments

2014-09-15T00:00:00+00:00

One of the fun benefits of running VsVim is that I’m constantly exposed to the amazing ways that vim can be configured. Many bugs in VsVim have to deal with commands that show up in vimrc files. Users are quick to share these files to help in tracking down the bug. The amount of customization that goes into these files is quite daunting and a reminder of just how flexible an editor vim really is.

But when I’m looking through these files I occasionally spot subtle bugs in the vimrc itself. Usually due to misconception in how various vim commands are implemented. One issue I see quite frequently is comments being added to key mappings. For example:

:nmap > >> " shorter shift command

Usually I’m a big fan of comments, even in script files, if they add appropriate context. The problem though is that these aren’t actually comments because key maps don’t have comments. From the help section map-comments:

It is not possible to put a comment after these commands, because the ‘”’ character is considered to be part of the {lhs} or {rhs}.

This means “shorter shift command” isn’t a comment but in practice it appears to act like one. The > key will now behave (virtually) identical to >>. It doesn’t seem to have any effect on the key mapping.

So if this isn’t a comment why is it acting like one? The reason is buried inside the same help page:

Note that when an error is encountered (that causes an error message or beep) the rest of the mapping is not executed.

In normal / visual mode the " key beigns a register sequence and must be followed by a valid register name. The <Space> key is not a valid register name which results in an error that causes vim to discard the rest of the key mapping. This particular error produces no messagge though, only a beep which many vmircs also disable.

This all combines to giving this the appearance of being a comment. In actuality though it’s just an invalid command.

The correct way to add comments to a key mapping is to put them on a separate line

" shorter shift commnad
:nmap > >>

Note: Earlier I mentioned that in the stated key mapping > behaved virtually identical to >>. In actuality it behaves like >><Space>. The space following >> is mapped and executed as the space motion (one column right)

Correctly inserting a new line in a VSIX

2014-09-02T00:00:00+00:00

To split a line in an ITextBuffer the developer simply needs to insert a recognized new line text into the existing line. Many VSIX extensions just default to Environment.NewLine for this task

_textBuffer.Insert(splitPosition, Environment.NewLine);

This is problematic because this is simply not the only possible new line text. The editor actually recognizes 6 varieties: '\r', '\n', "\r\n", '\u2028', '\u2029' and '0x85'. The other 5 sequences are more likely to come about on cross platform extensions. New files tend to end up with the preferred line ending of the host operating system (Windows is "\r\n" and OSX is '\n'). Hence cross platform extensions tend to end up with files that have a variety of endings.

Extensions which default to Environment.NewLine will work great right up until a cross platform developer starts using it. Then it will end up causing files to have multiple line ending sequences which quickly leads to all manner of tooling warniings: including Visual Studio popping up a modal dialog (yuck!).

To avoid this extensions should be careful to preserve the existing line ending text for a file. But where can an extension discover the correct line ending? The GetNewLineCharacter method looks attractive at a glance but is unacceptable for the following reasons:

It presumes that a buffer has a single new line text sequence for the entire buffer. This is generally true for a physical file but not true for an arbitrary ITextBuffer which can be a projection of multiple files.
Even in files which exclusively have a non-windows line ending it will still have the value "\r\n".

Instead the best way to choose the text for a new line is to just use the line break text from the line being split. Or in the case of the last line of the file, the line break text of the line above it. For example:

static string GetNewLineText(SnapshotPoint point)
{
  var line = point.GetContainingLine();

  if (line.LineBreakLength > 0)
  {
    return line.GetLineBreakText();
  }
  else if (line.LineNumber - 1 >= 0)
  {
    // If this is the last line then there is no line break, use the line above 
    var lineAbove = line.Snapshot.GetLineFromLineNumber(line.LineNumber - 1);
    return lineAbove.GetLineBreakText();
  }
  else
  {
    // Buffer only hase a single line, use the default new line sequence 
    return Environment.NewLine;
  }
}

Use this method instead of defaulting to Environment.NewLine, your cross platform customers will appreciate it.

Rethinking IEnumerable

2014-08-19T00:00:00+00:00

The .Net enumeration story based on IEnumerable<T> is a very succesful pattern. It’s the backbone of many different language and framework features including foreach, LINQ, iterators, etc … And yet when switching between C++ and C# and I’m often frustrated by its inefficiencs and quirks:

Accessing a single value requires 2 interface invocations: MoveNext and Current. Interface method invocation has extra overhead associated with it. Why pay that price twice instead of using a single method call?
It forces the allocation of a IEnumerable<T> value on the heap even when the enumerator could be implemented as a struct. Allocations in .Net are cheap not free and it’s frustrating to have one on such a core path.
The legacy of pre-generics .Net forces type safe collections to eventually implement the non-generic IEnumerable, IEnumerator and even IDisposable. I can’t remember the last time I actually used one of these and yet I have to write this boiler plate code every time I author a new collection.
Many collections, like List<T>, implement pattern based enumeration in part to avoid the above inefficiencies ¹. This is more code to write, test and maintain yet really doesn’t add any new features. At the same time it also complicates the ability of the developers to understand the mechanisms behind a given foreach block.

Recently as an experiment I decided to sit down and explore a new design for IEnumerable<T> which addressed these issues. In particular it had the following goals:

Use a single method call to advance the enumeration and access the next value
Use strongly typed generics
Do not force an allocation for the enumerator type
Have a single code path for enumeration
Dispose of all the legacy baggage ²

After some tinkering I settled on the following design:

public interface IEnumerable<TElement, TEnumerator>
{
  TEnumerator Start { get; } 
  bool TryGetNext(ref TEnumerator enumerator, out TElement value);
}

The most visible change here is the elimination of IEnumerator<T> in favor of an enumerator type parameter. This eliminates the forced allocation of enumerators and allows types like List<T> to use a more natural enumerator type like int.

The real advantage in this pattern is that it greatly simplifies the code necessary to implement enumerable:

class MyList<T> : IEnumerable<T, int>
{
  T[] _array;
  
  public int Start 
  { 
    get { return 0; } 
  }
  
  public bool TryGetNext(ref int index, out T value)
  {
    if (index >= Count) { 
      value = default(T);
      return false;
    }
    
    value = _array[index++];
    return true;
  }
}

This implementation is free from all of the boiler plate code necessary to implement .Net IEnumerable<T>. Instead it focuses on the actual code necessary to enumerate values. The full example of .Net enumeration is included at the bottom of this post. The code difference is staggering.

The code generation for a foreach over the new pattern is the following:

void M<TElement, TEnumerator>(IEnumerable<TElement, TEnumerator> enumerable)
{
  // Developer types 
  foreach (TElement current in enumerable) {
    // Loop body
  }
  
  // Compiler emits 
  TElement current;
  TEnumerator e = enumerable.Start;
  while (enumerable.TryGetNext(ref e, out current)) {
    // Loop body 
  }
}

This enumeration pattern for MyList<T> will now be consistent no matter the context in which it is enumerated: through MyList<T> or IEnumerable<T>. The enumerator type will now always be an int, it will always execute the same code path and this is the only enumeration code path to test.

The downside to this approach is that it makes the consumption of enumeration in generics more complicated: all methods and types now require 2 type parameters instead of 1. For advanced developers this isn’t a significant overhead but for the average .Net developer it’s a big increase in complexity.

The biggest problem though using this in practice is that it doesn’t work with existing foreach syntax. Hence all loops needed to be typed out by hand and the actual looping pattern is not a great pattern for humans to be dealing with. That would obviously go away if this was a first class language construct.

Overall I’m pretty happy with this design though. Maybe I’ll take a stab at prototyping Roslyn support for this feature at some point to get my foreach problems solved.

As promised before, this is what the MyList<T> implementation of .Net IEnumerable<T> would look like (55 lines of code vs. 19 for the new pattern).

// Old Enumeration Pattern 
class MyList<T> : IEnumerable<T>
{
  T[] _array;

  internal struct Enumerator : IEnumerator<T>
  {
    int _index;
    MyList<T> _list;

    internal Enumerator(MyList<T> list)
    {
      _list = list;
      _index = 0;
    }

    public T Current
    {
      get { return _list._array[_index]; }
    }

    public void Dispose()
    {

    }

    object System.Collections.IEnumerator.Current
    {
      get { return Current; }
    }

    public bool MoveNext()
    {
      if (_index + 1 < _list._array.Length)
      {
          _index++;
          return true;
      }

      return false;
    }

    public void Reset()
    {
      _index = 0;
    }
  }

  public IEnumerator<T> GetEnumerator()
  {
    return new Enumerator();
  }

  System.Collections.IEnumerator System.Collections.IEnumerable.GetEnumerator()
  {
    return new Enumerator();
  }
}

In the pre-generics world of .Net pattern based enumeration also had the added benefit of type safe enumeration ↩
Eliminating legacy is rarely a reality but it’s fun to think about ↩

Testing for VSIX Memory Leaks

2014-07-30T00:00:00+00:00

A common bug in VSIX projects is to hold onto an ITextView instance long after it has been closed. This is problematic because it ends up preventing a large number of resources from being collected including the ITextBuffer, language service elements, WPF items, other extension objects, etc … In short it is a substantial memory leak.

The vast majority of these leaks occur with the following pattern:

A VSIX MEF component hooks into an event that has a lifetime not tied to a particular ITextView instance. For example: an extension listening to a settings changed event
The event handler transitively contains a reference to an ITextView.

Instant memory leak.

This problem is not specific to VSIX but more of a general issue with .Net event handling. MEF tends to maginify this problem because it makes it so easy to connect components with very different lifetimes. Just slap a [Import] ISettingsService on a field and that global service is now available for use in any context.

Spotting these leaks is difficult at best even in a moderately sized extension. MEF hides so much of the interaction between components that it’s often hard to understand how the lifetimes relate to each other. The only real way to avoid this problem is to test for it.

Testing for a memory leak is actually rather straight forward:

Create the object you are concerned about leaking and save a copy in a weak reference
Run the scenario
Clear all strong references to the object in the test code and run the GC a few times
If the weak reference still has a value then the scenario leaked memory

To apply this pattern to a VSIX project it means creating instances of ITextView. The only realistic way to do this is to create a MEF container which has both the Visual Studio WPF editor and the necessary parts of your extension. Doing this by hand is rather tricky. A much easier way is to leverage the EditorUtils project. It is available on NuGet and has APIs for doing exactly this.

using EditorUtils;
...
var editorFactory = new EditorHostFactory();
editorFactory.Add(new AssemblyCatalog(typeof(MyExtension).Assembly));
var editorHost = editorFactory.CreateEditorHost();
var textView = editorHost.TextEditorFactory.CreateTextView();

The EditorHostFactory type is responsible for creating the MEF container. The editorFactory.Add line is simply inserting the assembly of VSIX extension being tested into the container. The resulting EditorHost instance wraps a MEF container with both the VS editor and the targetted extension. Now extension elements will be created exactly as they are when running in Visual Studio. This makes writing tests extremely easy.

[Fact]
public void Scenario()
{
  ITextView textView = CreateCSharpTextView();
  textView.TextBuffer.Insert(0, "hello world");
  textView.Close();
}

The Insert line naturally needs to be replaced with some code which excercises your extension. Also this is relying on the test being within the larger test template outlined below. The first reaction you may have when looking at this template is:

Holy crap! That’s quite a bit of boiler plate code!

Yes indeed it is. Unfortunately it’s all necessary to ensure the leak detection is done properly. Good news though is the code is copy, paste and forget. The actual test logic is mostly unaware of this pattern (and it can be abstracted out to a base type if that suits your needs better)

Overall this may seem like an onerous process to go through. But let me assure you that the results are worth it. Early versions of VsVim were plagued with memory leaks. After tracking down a particularly nasty one for the umpteenth time I decided to take the time to work through these tests. They have saved me from introducing new leaks countless times since then.

The test template

public class MemoryLeakTest : IDisposable
{
    readonly EditorHost m_editorHost;
    readonly List<WeakReference> m_textViewList = new List<WeakReference>();

    public MemoryLeakTest()
    {
        var editorFactory = new EditorHostFactory();
        editorFactory.Add(new AssemblyCatalog(typeof(MyExtension).Assembly));
        m_editorHost = editorFactory.CreateEditorHost();
    }

    public void Dispose()
    {
        RunGarbageCollector();

        foreach (var weakReference in m_textViewList)
        {
            Assert.False(weakReference.IsAlive);
        }

        // Don't let the GC collect the MEF container and hence hide the leaks 
        GC.KeepAlive(m_editorHost);
    }

    static void RunGarbageCollector()
    {
        for (var i = 0; i < 15; i++)
        {
            // Got to clear out any lingering WPF actions which may hold onto the ITextView
            DoEvents();
            GC.Collect(2, GCCollectionMode.Forced);
            GC.WaitForPendingFinalizers();
            GC.Collect(2, GCCollectionMode.Forced);
            GC.Collect();
        }
    }

    static void DoEvents()
    {
        var dispatcher = Dispatcher.CurrentDispatcher;
        var frame = new DispatcherFrame();
        Action<DispatcherFrame> action = _ => { frame.Continue = false; };
        dispatcher.BeginInvoke(
            DispatcherPriority.SystemIdle,
            action,
            frame);
        Dispatcher.PushFrame(frame);
    }

    ITextView CreateCSharpTextView()
    {
        var contentType = m_editorHost.GetOrCreateContentType("csharp", "code");
        var textView = m_editorHost.CreateTextView(contentType);
        m_textViewList.Add(new WeakReference(textView));
        return textView;
    }
}

Deploying a VSIX Project to AppVeyor

2014-06-07T00:00:00+00:00

As usual Scott Hanselman wrote a blog post that got me super excited about a new piece of software: AppVeyor. A continuous integration system which had built in integration with github. I couldn’t wait to try it out on a few projects.

Getting Builds Working

The blog post promised a simple deployment story but I was still skeptical. Many of my OSS projects are VSIX projects (Visual Studio Extensions). These are notoriously difficult to get building in a clean environment because of their dependency chain

VSIX production depends on several MSBuild Tasks and Targets
Those targets depend on having the full Visual Studio SDK installed
The Visual Studio SDK depends on having Visual Studio Professional or higher installed

That last dependency is usually a killer. Who can really afford ¹ to put a full copy of Visual Studio on every virtual machine they farm out for a continuous integration system? The best most systems can do is install the various Express SKUs on the machines which isn’t enough to install the SDK.

Fortunately this is a problem that AppVeyor had already run across. Their solution was simple: forgot the SDK installer, just xcopy deploy the necessary SDK bits to get VSIX builds working. It’s simple, brute force and most importantly, it works! Well almost, there is one small change that needs to be made to the projects: deployment on build must be disabled. The AppVeyor setup simply doesn’t support this. Turning this off requires only a one line change to the csproj.

<DeployExtension Condition=" '$(AppVeyor)' != '' ">False</DeployExtension>

Once I made that small change my builds were up and running. Literally nothing else had to change. Pretty sweet.

Getting Tests Working

Visual Studio is still a 32 bit application. This dependency extends even into a set of the managed DLLs that VSIX projects depend on. As such my unit tests only properly function when run in 32 bit mode.

All of my tests are written in xunit and right now AppVeyor only has support for running the 64 bit version. This was causing all of my unit tests to fail out of the box. Working around this is easy enough though. Just check in a copy of the x86 xunit runner to your repository and change the appveyor.yml file to manually invoke the tests.

test_script:
  - Tools\xunit.console.clr4.x86.exe Test\VimCoreTest\bin\Debug\Vim.Core.UnitTest.dll /silent
  - Tools\xunit.console.clr4.x86.exe Test\VimWpfTest\bin\Debug\Vim.UI.Wpf.UnitTest.dll /silent
  - Tools\xunit.console.clr4.x86.exe Test\VsVimSharedTest\bin\Debug\VsVim.Shared.UnitTest.dll /silent

I alerted AppVeyor to this problem and are looking into it.

Overall I’m really happy with using AppVeyor as the CI system for my OSS projects. The setup is easy, the team is very responsive to questions and customizing the environment is very straight forward.

Except Microsoft ↩

We should ban the phrase "thread safe"

2014-05-22T00:00:00+00:00

There few are phrases in programming that make me shudder more than

This type is thread safe

Reading this phrase is like hearing nails on a chalk board. It often makes me physically cringe.

The phrase thread safe makes it sound like thread safety is on / off property of a type. Nothing could be further from the truth. There is a wide variety of multi-threaded usage scenarios for types that simply cannot be decribed in a binary fashion. Describing it in such a way serves to do nothing other than confuse users.

A much better way to describe the thread safety of a type is to enumerate the scenarios in which the type can or can’t be used. Giving users concrete information on how the type will behave in specific circumstances gives them a much better chance of correctly using the type in their own programs.

Even better once the scenarios are enumerated you’ll often find these types fall into exitsing well known patterns which the user is more likely to have experience with. This gives them a much better chance of succesfully using the type

Here is a non-exhausting sampling of the most common multithreaded patterns I encounter in my day to day programming

Data Race Free

A type which is data race free will never corrupt its internal state no matter how many threads are reading from and writing to it. Sharing such a type between threads is safe in that the internal data structures will never be corrupted. A data race free list for example will never have a count different than the actual number of elements stored inside it.

Examples of data race free types are this include Java vectors, .Net 1.0 synchronized collections, and .Net 4.0 concurrent collections.

Such a type is still fraught with usability problems though. For example here is an incorrect sample using a synchronized version of ArrayList

static object GetFirstOrDefault(ArrayList synchronizedList) { 
  if (synchronizedList.Count > 0) {
    return synchronizedList[0];
  }

  return null;
}

None of the individual statements here are an incorrect usage of synchronizedList yet the function itself is wrong. It’s possible for any number of threads to execute in between the initial if block and the return statement. Any of which could clear the list thus causing an execption on the return statement ¹.

Multiple Reader, Single Writer

This class of types can be safely read from mulitple threads or written to by any single thread. However these operations cannot be done at the same time or the data structure will possibly corrupt its internal state.

This pattern allows for useful parallel operations that simply need to read from a type. For example projecting the contents of one collection to a new one.

In my experience this is the most common pattern for types because it is the natural default. There is simply no danger in reading the same memory from multiple threads so long as it is consistent for the duration of the reads. This is how properties and query methods naturally behave. Unless a type goes out of its way to silently mutate on reads or has a particular thread affinity it will function in this manner

Thread Affinity

Thread affinitized types can only be safely accessed from a specific thread in the program. Any attempt to access instances of the type from code executing on a different thread is an error. Such code must somehow transfer execution to the correct thread in order to safely inspect the object.

This pattern is probably most common in UI frameworks. It is in fact the source of one of the most frequently asked questions on stackoverflow.

Why does an exception get thrown when I change a property on my control?

var worker = new BackgroundWorker();
worker.DoWork += DoWork;

private void DoWork(object sender, DoWorkEventArgs e) {
  /* background caclulation */
  _theLabel.Text = theResult;
}

All WinForm controls are affinitized to the UI thread. Any attempt to modify them from another thread will be met with an exception. In this case the BackgroundWorker callback executes on a different thread and hence this is a violation of the types threading contract.

Lucikly Winform types are one of the few types that enforce their thread safety contract at runtime.

Not Even Read Safe

The ability to safely read from a type on multiple threads depends on the read operations either a) being non-mutating or b) having internal, and potentially costly, synchronization around the mutations. Types which do neither of these simply can’t be accessed in this manner.

This may seem rather strange but it’s actually something the majority of C# users consume every day: LINQ queries. An IEnumerable produced from a LINQ query aren't actually evualuated until they are read from. Queries like Distinct must use a table in order to filter out previously returned values. Hence every read operation is reading from and potentially writing to the table. This mutation disqualifies it from being succesful queried from multiple threads

Less well known but particularily interesing are types like Splay Trees. This binary search tree which optimizes lookups for recently accessed nodes. It does this by mutating the tree to push recently accessed nodes closer to the root on read operations (mutation on read).

Immutable Types

This is the one class of types to which thread safe actually can be validly applied. A type which never changes satisfies any rational expectation a developer has for safe use amongst multiple threads. Seemingly random operations like thread ordering have no effect on the outcome of operations on an immutalbe type

But if you have an immutable type calling it as such is far more descriptive than thread safe. It’s a rock solid guarantee that exists through the lifetime of the program. Don’t bother with ambiguous terminology when far more descriptive alternatives exist.

This wide variety of scenarios is why the term thread safe simply needs to be banned. It is an ambiguous description of a type that serves no purpose other than to cause confusion.

Don’t ever let another program get away with saying thread safe. Call out this ambiguous description of a type and ask them to enumerate the specific threading scenarios for the type. And most importantly, once they enumerate the details make sure it is documented in the type itself and not just an email.

For the curious I already covered this problem in detail in a previous post ↩

The Great Naming Hack

2014-05-12T00:00:00+00:00

Inheritting a legacy code base is a rite of passage for developers. This is the event which takes you from the mentality of clean, documented, tested code you wrote during university into the ugly real world of compromises. It is very much a “how the sausage is made” moment

My first experience with this was transitioning to the languages team in Visual Studio back in 2006. The C++ code base I inheritted was mostly written from 1998 onwards but had significant portions which were much, much older. There were no enforced standards at the time hence developers checked in whatever style or practice they wished

As you can imagine that kind of environment creates quite a few humorous gems over time. Even after years of working there we’d still find new ones every few weeks. The one that will always stick with me though is the great naming hack. It is best exemplified by this sample

struct SymbolNode : ExprNode
{
  union
  {
    ExprNode*  pnodeQual;
    ExprNode*  BaseReference;
  };

  union
  {
    NameNode* pnamed;
    NameNode* Name;
  };
};

Typically a unions are used when two mutually exclusive pieces data are present in the same data structure. It uses a single storage location for all the values hence removing wasted space. When I first encountered this my thought was

How could a union possibly be relevant here? All of the members are of the same type, they just have different names.

Luckily one of our developers was on the team when this was introduced and was able to provide the history of this change. At some point in the past the team had entered into one of the great religous programming debates

Hungarian Notation or No Hungarian Notation?

Unfortunately they were not able to reach a concencus. Instead the outcome was to let developers use whichever they liked in the files they owned. As a warning to others files were marked with comments of “hungarian notation only” or “no hungarian notation” at the top. But they still struggled on how to handle data structures shared between files. Until that is one developer had an epiphany

We can use a union to create multiple names for the same field. This way every field can have a hungarian and non-hungarian name!!!

The result is pretty much the chaos you are probably envisioning right naw. Many files used ‘pnodeQual’, others used ‘BaseReference’ and anyone who crossed between such files had to alternate between them based on the style of the current file. It was … difficult to deal with.

Unlike many legacy code stories though this one has a happy ending. After learning the history of this construct the team quickly decided to abolish it. We took a vote, decided against hungarian notation and spent the time necessary to delete the unions from the shared data structures.

Round Trippable VSIX Project Templates

2014-05-02T00:00:00+00:00

A short time ago I wrote a post about how to turn a standard VSIX project into one which could be round tripped into any version of Visual Studio. This set of changes also fixed other issues like debugging + SCC, assembly binding, etc … I got a lot of positive feedback and nice links to projects that developers upgraded as a result of my post.

While this was working great for developers with existing projects it still amounted to routine extra work for developers creating new VSIX projects. They either had to

Copy the RoundTripVSIX Source locally and do a bunch of renaming
Create a new VSIX project and mirror the exact edits listed in the previous post

A much better solution would be for me to publish new VSIX project templates that are round trippable to the gallery. With that developers can just do the normal File -> New Project -> Round Trip VSIX and be up and running immediately.

Originally I didn’t do this because authoring and packaging project templates was a rather involved effort. I just didn’t have the time necessary to learn it well enough to ship a good product. Recently though I was reading about the Side Waffle project. Its promise of making project / item template creation as easy as creating the project was very appealing.

Yesterday I had an hour of free time and decided to take Side Waffle for a spin. Man did it deliver on its promise. I installed the Side Waffle extension, watched a 5 minute tutorial and had my first version of the templates running just a few minutes later. Their promise of easy template creation was spot on. I frankly have no idea how the underlying packaging mechanism works but I’ve succesfully created a number of templates at this point (basically the opposite of what I expected).

The fully round tripping templates are now available on the Visual Studio Gallery and in source form. Contributions and suggestions are always welcome!

VSIXTemplates on Visual Studio Gallery
VSIXTemplates source on Github

Interviewing College Candidates

2014-01-06T00:00:00+00:00

Lately I’ve been reading a lot about peoples interview processes and it inspired me to share my process for interviewing college candidates. I’ve been doing interviews at Microsoft for ~10 years now and developed this process over that time. The format for the interviews are typically 1 hour with just me and the candidate. Usually in my office or occasionally a conference room. Unfortunately 1 hour is a very short amount of time in which to judge a candidate and I try to combat that by planning ahead and working with the below process.

Note that this post is specifically about how I interview college candidates or candidates with a small number of years in the industry. I have a much different process for senior level developers (which is more of what I do these days).

Pre Interview Setup

The candidate deserves my undivided attention during the interview. Before I go out to greet the candidate I turn my phone off and put it in my desk, close my email, turn off the speakers to my computer and lock the screen. The only visible distraction I have is a clock to make sure I’m keeping the interview moving along.

The Introduction

My very first goal in an interview is to make sure the candidate is comfortable in the environment. Interviewing is a nerve racking experience for some people (especially college candidates who are quite possibly doing this for the first time ever). I’m rarely the first person on an interview loop and hence I often have to deal with candidates that are tired or perhaps rattled a bit by an interview earlier in the day. I find people perform best when they are in a relaxed environment and I do my best to get them there.

One technique I’ve found that helps relax people is to get them talking about something they are passionate about. I get every candidates resume a day or two before the interview and I look over it in detail. I’m looking for anything that they have put significant time or effort into: a senior project, GitHub contributions, stack overflow, etc ‘

Once I find that project I spend 30 minutes or so researching it myself. My goal is to get enough of an understanding of the project that I can have a conversation about it. I also write down some really easy questions about the project. Presenting the candidate with questions on subjects they are passionate about is a good way to get the candidate relaxed and give them an early confidence boost. It’s much better then starting them off with questions I’m passionate about

I usually start off with the standard small talk. Who I am, how long I’ve been here and what groups I’ve worked in at Microsoft. I ask them the same about themselves and then transition into the research I’ve done on them

I was looking over your resume and noticed your senior project on improving TCP startup. Tell me about that.

Some people are happy to go on at length here and some are a bit more reserved. For the reserved I prompt them with the questions I’ve already written down.

Why is TCP intentionally slow at startup?

Your project focused on improving algorithm X, what was wrong with it?

Remember the goal here is to relax the candidate and increase their confidence. Take their project, craft easy questions for it based on the work their project said they did. Let them shine here.

I generally reserve 10-15 minutes for this section of the interview

The Technical Interview

I start the technical portion off with a very easy and vaguely worded question. My favorite question is the following

Write a function that takes a collection of letters and determines if there are any duplicate letters

The candidate is free to constrain this problem however they like

Can I use a hash table? Sure

Can I assume the letters are all ASCII? Sure

Can I assume they come in sorted order? Sure

Can I sort them? Sure

Can I use something like bubble sort to check every combination? Sure

I do this for a couple of reason. In part I want to keep the candidate comfortable and give them an easy win on the white board. I don’t care how they solve it, just that they can solve it. It’s a modified fizzbuzz problem that tells me they can at least write code in the language of their choosing. The other reason I do this is because I’m curious to see what questions they will ask (if they ask any). What they ask often reveals a good deal on how they approach coding.

After they get the solution correct and we walk through it I start taking things away.

That hash table makes it easy doesn’t it? How would you solve it without a hash table?

ASCII really makes the problem easy. How would you solve it with Unicode characters?

Could you solve this without allocating any memory?

I make subtle changes to the question that make the solution incrementally harder and gauge how they react to the changes. I do want to stress incrementally here. I don’t go from easy to hard with a single twist. Instead I add constraints to gradually and incrementally up the difficulty.

It’s not critical that they solve every twist I give them. Really what I’m after here is how they think about coding and whether or not they can adapt their solution to changing requirements (aka my life as a programmer). I’m also curious how they react on a personal level to the changes. Do they get angry, excited, etc ‘

Throughout this section of the interview there is one thing I’m constantly vigilant about

Do not ever let the interview session go stale for more than 5-10 seconds

There is 0 value in having an interview where me and the candidate are just staring at each other silently for several minutes. If they get stuck on a problem it is my job as the interviewer is to help get them unstuck. Ask them what they’re stuck on, what they’ve considered, get them to draw out sample input, anything that gets them talking or writing on the white board.

My time reservation for this section is 35-40 minutes but I’m a bit more flexible. If the candidate is a clear hire I usually stop earlier and spend more time on selling, answering questions. If a candidate is struggling I give them as much time as possible to get to a good solution.

The Wrap Up Conversation

In general I let the interview wrap up be very candidate driven. I open the floor for questions they have. If they don’t have any I try and answer the most common questions

What is the environment in my group like
What are the ups and downs of my group
What is life in Seattle like
Why did I pick Microsoft over company X

It is really important in the wrap up to make sure the candidate leaves feeling good about the interview. Even if the candidate completely and utterly screwed up the interview I try to make sure they leave feeling good about the session. They could have just had a bad day, you may have just asked them the one question they couldn’t answer. It’s very possible that you are the only “No Hire” they will get all day long. Or maybe they’re not good enough now but a year from now they’ve gotten enough better that they nail the interview and you end up working with them.

To sum it up in one statement

Never let a candidate leave feeling bad about themselves.

Round Tripping a VSIX Project

2013-12-04T00:00:00+00:00

Visual Studio 2012 introduced project file round tripping feature. This lets developers edit the same project in Visual Studio 2010, 2012 and 2013 without the need to upgrade the project file or modify it in any way. This was a highly requested feature by customers that allowed them to edit their project no matter what version of Visual Studio they had on their machine. The previous forced upgrade model for new versions was a big adoption blocker. Now customers can just grab the latest Visual Studio version and start hacking away

Unfortunately this feature does not work on all project types including VSIX projects. This meant that Visual Studio Extension authors who wanted to developer extensions for all versions of Visual Studio were essentially locked into editing that code with 2010.’? This is doubly unfortunate given that extension authors are the type of developers most likely to early adopt new versions of Visual Studio.

The good news though is that it is possible to have a VSIX project be round tripped, it just requires a bit of special sauce to get working. Once done though a VSIX project can be fully edited, F5’d, etc ‘ in any version of Visual Studio.

Originally this blog post was going to be a step by step process for getting a 2010 VSIX project into full round tripping mode. However as I started to write the post it was looking rather dull. Virtually every line was a variant of

Open File <…>
Paste on line <...>

After pasting some rather ugly MsBuild XML into my post I realized that there was a much better way to convey this information: actually demonstrating the transition from a 2010 extension to a fully round trippable one on a real project. Essentially I decided the best way to do this was to speak in code.

Hence I created a GitHub project that starts with a simple 2010 extension and step by step (or commit by commit) turns it into a fully round trippable one. Every commit has detailed comments about why the edits were taken, what features they provide and what limitations remain. The full transition is available here

GitHub Project: https://github.com/jaredpar/RoundTripVSIX
Commit Log: https://github.com/jaredpar/RoundTripVSIX/commits/master

I hope that this speaks better to other extension developers out there and helps them get to a happy round trippable world.

Immutable isn't just for parallel code

2013-07-02T00:00:00+00:00

For the last 6 months the BCL team has been hard at work shipping an out of band release of immutable collections for .Net. Most recently delivering an efficient implementation of ImmutableArray

http://blogs.msdn.com/b/dotnet/archive/2013/06/24/please-welcome-immutablearray.aspx

Unfortunately with every announcement around these collections I keep seeing the following going past in my twitter feed.

That looks neat. Next time I’m writing parallel code I’m going to try that out.

No, no no!!! Immutable types are not just for parallel code. They are very useful in even single threaded applications. Developers are missing out by thinking of them as a parallel only construct. Any time you want to enforce that the contents of a collection never change you should consider an immutable type.

The Provider

Consider the case of an object which provides a collection to callers where the contents never change. Something like Assembly.Modules. A type like Assembly must be robust in the face of misuse by any caller and always present the same set of Module values. Given this constraint what type should it use though for the property?

It can’t do something as simple as using a List with a similarly typed backing field. Returning such a value would allow a devious caller to clear the list and spoil the results for everyone else. It cannot even store a List internally and return an IEnumerable as anyone could just come along and down cast to List and clear the collection.

Assembly assembly = ...;
// Evil laugh
((List<Module>)assembly.Modules).Clear(); 

Instead it chooses to be robust by returning a freshly allocated array on every single call to Modules ¹. This works but is a very wasteful process and results in many unnecessary allocations.

If this API were being designed today this would be a perfect candidate for using ImmutableArray. This value can be safely stored and returned with no fear of the caller deviously mutating the result. There is simply no way of doing so.

The Consumer

Now consider the case of the consumer who wants to store a collection of Modules instances and do multiple lazy independent calculations on them. In order for the different calculations to be correct they need to ensure the collection of Module instances don’t change from operation to operation. Hence they have to make a decision when storing the collection in the constructor

class Container {
   IEnumerable<Module> m_modules;

   public Container(IEnumerable<Module> modules) { 
     // Do I trust my caller' 
     m_modules = modules;
   }
}

The constructor can choose to do one of the following

Create a private copy of the input collection that it doesn’t mutate
Create no copy and hope that the caller never mutates the input collection

The first option is wasteful and the second is just a bug waiting to happen a year from now when someone decides to reuse a List for another purpose. With immutable types the container has a much better third option: demand a collection that never changes

class Container {
   ImmutableArray<Module> m_modules;

   public Container(ImmutableArray<Module> modules) { 
     // Trust no one 
     m_modules = modules;
   }
}

The callee has now forcefully stated to the caller exactly what type of data it expects. It no longer has to make a wasteful copy or hope for good behavior. True this may force the caller to create an immutable copy of the value it holds. It’s also just as likely that the caller will be in a position to provide the collection without any copies. If it takes the collection as input it can simply pass along the requirement in its parameter list. Or if it is the original creator of the collection it can do so as an ImmutableArray from the start and avoid the extra copy altogether. Over time, code bases which are assertive about using immutable collections will see a decrease in allocations because they will feel more comfortable with sharing data between independent components.

This is just a small sample of cases where immutable collections are useful in day to day code. The more you use them the more uses you will find for them. At some point you may even find yourself asking the following question when writing up a type

Do I actually need to mutate this collection after I finish building it?

Generally speaking the answer to this is no. And this is why you should be using immutable types.

If you dig deep into the implementation you’ll find it’s actually a fresh array of RuntimeModule[]. So even though they allocate a new array on every call you can’t safely write Module instances into it unless they happen to be instances of RuntimeModule. So wasteful! ↩

Don't mix await and compound assignment

2013-02-11T00:00:00+00:00

The 5.0 release of C# introduced the await keyword which makes it extremely easy to use Task in a non-blocking fashion. This allows developers to replace either blocking calls to Task.Wait() or complicated combinations of ContinueWith and callbacks with a nice simple, straight forward expression

Task<int> task = ...;
int local = await task;
Use(local);

What most people don’t consider is the implications of this non-blocking support. At the point the ‘await task’ expression executes the program can do one of two things. If task is resolved the await expression will complete and the method will continue executing immediately. If it’s not then the method will pause and some other part of the program will wake up and start running. This other code interleaves with with the execution of this method. This means subtle timing changes in when Task values are resolved can drastically alter the order in which a program executes.

These method interleavings can be the source of many subtle bugs in the program. They are typically timing related and hence often don’t predictably reproduce and possibly don’t show up at all until the program is executing on a customers machine. One of the most common bugs I see is when developers mistakenly combine compound assignment with the await keyword

x += await y;

The C# compiler will rewrite this code into roughly the following ¹

x = x + await y;

And it executes in the following steps

load x onto the stack
await y
Push result of ‘await y’ onto the stack
Add the stack values
Store into x

In the case where ‘y’ isn’t resolved the execution of this code will stop at step #2. At which point some other code will begin executing in its place. If ‘x’ is a local there isn’t much danger here but what if ‘x’ represents a field that is accessible to another part of the program’ And what if that field is modified while this expression is paused at #2’ When this statement resumes it will never re-read the value of ‘x’ and hence any writes to it which occurred during the pause will be erased once the expression completes. Or in other words, it will quite simply ignore the other write.

I most frequently see this problem with accumulator scenarios. A collection of tasks are spun up and the results are tallied up as they complete

class Accumulator
{
    private int m_sum;

    public int Sum
    {
        get { return m_sum; }
    }

    public async Task Add(Task<int> value)
    {
        m_sum += await value;
    }
}

This code is fundamentally incorrect because it invites this very problem. Consider the following scenario

Call to Add is made with an unresolved Task. Add pauses on the task having already read m_sum onto the stack
Call to Add is made with a resolved Task with a value of 4. Add completes and m_sum is now 4
Task from step 1 resolves with a value of 2. The value on the stack for m_sum is still 0 so m_sum is written out as 2 instead of 6

The way to avoid this problem is to simply not mix await and the compound operator. Instead store the await value into a temp and then do the assignment without the risk of interleavings.

public async Task Add(Task<int> value)
{
    var temp = await value;
    m_sum += temp;
}

Here is a sample which will demonstrate the bug in a deterministic fashion.

var accumulator = new Accumulator();
var taskCompletionSource1 = new TaskCompletionSource<int>();
var taskCompletionSource2 = new TaskCompletionSource<int>();

var task1 = accumulator.Add(taskCompletionSource1.Task);
var task2 = accumulator.Add(taskCompletionSource2.Task);
taskCompletionSource2.SetResult(3);
taskCompletionSource1.SetResult(2);

await task1;
await task2;
Console.WriteLine(accumulator.Sum);

This code will print out 2 instead of the expected 5

I use ‘roughly’ here because it the compiler actually does a more complicated rewrite. It ensures that the side effects of ‘x’ happen exactly once during the execution of this method. For locals though this is roughly the code that is generated and serves fine for this example ↩

Not all "true" are created equal

2012-08-28T00:00:00+00:00

During a review of some low level bit manipulation logic a developer raised a question about the correctness of a piece of code which allowed any arbitrary byte to be seen as a bool. No one could recall if true was defined as not 0 or simply 1. If it was the latter then the code was allowing for a large range of invalid bool values to be created. A quick look at the CLI spec revealed the immediate answer (partition III section 1.1.2)

A CLI Boolean type occupies 1 byte in memory. A bit pattern of all zeroes denotes a value of false. A bit pattern with any one or more bits set (analogous to a non-zero integer) denotes a value of true.

A quick test backed up this particular assertion. Any non-zero value is true and 0 is indeed false. We took the test one step further and discovered, to the surprise of about half of us, that just because two bool values are true, doesn’t mean they’re equal.

class Program
{
    [StructLayout(LayoutKind.Explicit)]
    struct Union
    {
        [FieldOffset(0)]
        internal byte ByteField;

        [FieldOffset(0)]
        internal bool BoolField;
    }

    static void Main(string[] args)
    {
        Union u1 = new Union();
        Union u2 = new Union();
        u1.ByteField = 1;
        u2.ByteField = 2;
        Console.WriteLine(u1.BoolField); // True
        Console.WriteLine(u2.BoolField); // True
        Console.WriteLine(u1.BoolField == u2.BoolField); // False
    }
}

I was one of those who was surprised! :)

Authoring a Utility Library for Visual Studio

2012-05-07T00:00:00+00:00

As I’ve developed VsVim over the years I’ve authored quite a few reusable Visual Studio components. For the last 6 months I’ve had many of these factored out to a separate utility library and this last week I decided to publish them as a separate NuGet package. Even if no one else every uses the library I want to reuse the utilities in other projects I’m working on and NuGet is the perfect distribution mechanism. For those interested I’ll be blogging about these components and why I authored them in the coming weeks (hint: perf, perf and more perf). I wanted to blog about my the rules I learned from this exercise because even as a seasoned extension author I hit a couple of very surprising problems along the way. Hopefully the lessons I learned will help out the next person to attempt this

Utility libraries must be strongly named

This is an unfortunate truth of authoring a utility library in Visual Studio. If you intend to ever release more than one version of a utility library then you must accept that there will be two extensions referencing different versions of your library in the same instance of Visual Studio. Unless the assemblies are signed the CLR will only load one version into the AppDomain. This means one extension will see the version it expects and the others won’t. This would be disastrous if the newer version of the library had new MEF interfaces, features, etc … Simply being very diligent with version numbers just isn’t enough here. When comparing a DLL reference to a DLL loaded in memory the CLR will ignore version numbers on unsigned assemblies ¹. If an app has a reference to MyUtility.dll at Version 99 and MyUtility.dll at Version 1 is loaded the CLR will consider it a match. There’s nothing I’m aware of, other than strong names, that will change this behavior.

This also means you can’t even rely on the latest version of your utility library being loaded. Extension load order is not defined in Visual Studio ². Hence it’s quite possible that extension referencing the oldest version of your utility library loads first and establishes that version as the one every other extension will use.

When an assembly is strongly named the CLR will respect version numbers and load multiple versions of the assembly into the AppDomain at the same time. Every extension will then see the version they are expecting independent of what other extensions are installed on the machine.

MEF isn’t version safe by default

MEF gives the appearance of being a model that deals in terms of types. Contracts are typically defined in terms of interfaces, [Export] of the implementation use a typeof experession and the associated [Import] is tagged on a type location. Everything about it screams Type, yet MEF doesn’t actually deal in terms of types, it primarily deals in terms of strings. Specifically in the form of a contract name and type name.

Every export and import have a contract name and type name associated with them. Only when both the contract name and type name match does MEF consider an Import and Export to match. Consider:

[Export(typeof(IObjectCache))]
internal sealed class ObjectCache : IObjectCache {

}

The above code did not create an export of the .Net type IObjectCache or even it’s assembly qualified name. Instead it created an export with

Contract Name: SomeNamespace.IObjectCache
Type Name: SomeNamespace.IObjectCache

Note: If you don’t specify a contract name explicitly MEF will just reuse the type name it generates

Notice that no assembly information is captured in this export. It’s just a type name plus the enclosing namespace. This contract will match up with any other Export of a type with the same fully qualified name as IObjectCache in any assembly loaded into the AppDomain. And this is exactly what will happen if you have multiple versions of your library loaded into the Visual Studio process ³

In order to have version same MEF components the export and import contracts need to be different for different versions of your library. The easiest way to achieve this is to embed the assembly version into the contract name portion of an Export and Import. In my projects I achieve this by means of a constant which I reuse in my AssemblyVersion attribute and Exports

public static class Constants {
    public const string AssemblyVersion = "1.0.0.0";
    public const string ContractName = "MyUtility " + AssemblyVersion;
}

[Export(Constants.ContractName, typeof(IObjectCache))]
internal sealed class ObjectCache : IObjectCache {

}

[assembly: AssemblyVersion(Constants.ContractName)]

Unfortunately this messiness isn’t something that can be self contained within your library. It’s a price you must push down to your consumers as well. Their [Import] attributes must have the same contract name else MEF won’t consider them a match and will reject the composition. Hence consumers of your library must use a similar pattern.

public class MyService {
    [ImportingConstructor]
    public MyService([Import(Constants.ContractName)] IObjectCache objectCache)
    {

    }
}

This applies to all ImportingConstructor, Import and ImportMany usage of your types.

Both of these lessons set me back quite a bit. But eventually I was able to produce the EditorUtils package I’d been working on and change VsVim to use it. Hopefully there aren’t many more surprises waiting for me around the corner

Except for Silverlight ↩
Unless there is an explicit entry in the manifest file declaring a dependency. Won’t ever happen for unrelated extensions. ↩
If this surprises you then you’re not alone. I frankly disbelieved the first person who told me this and had to create a sample app to prove it to myself ↩

VsVim 1.0 Released

2011-07-01T00:00:00+00:00

I just released an update to VsVim for Visual Studio 2010. This is available on the extension manager in Visual Studio or can be downloaded directly at the following link. ¹

Link: http://visualstudiogallery.msdn.microsoft.com/en-us/59ca71b3-a4a3-46ca-8fe1-0e90e3f79329

GitHub: http://github.com/jaredpar/VsVim

**What does 1.0 mean? **

Several people asked me what 1.0 meant for VsVim’ From the first release of the project to the web I’d envisioned 1.0 as the point where I had a highly functional Vim implementation in Visual Studio.’? Something a Vim user could add to Visual Studio and become immediately more productive. Given the feedback I’d been receiving and seeing the trend in filed issues from major features to more minor features and bugs I felt like VsVim met that goal.

For me personally it’s a significant milestone in what started out as a pet project to do little more than learn F# and the new Visual Studio Editor APIs. It was only after a few months of toying around in my spare time that I thought it might end up having value to people other than myself. Even then I was just as interested in publishing a working Visual Studio 2010 editor extension for documentation purposes as I was in the actual functionality (although my primary goal quickly transitioned towards functionality). I thought it would be great if I ended up with 1000 downloads in the first year. As of this writing VsVim is at 24,000 downloads and lots of positive community feedback!?? It all really exciting for me.

Release Notes

This release is mainly a bug fix release on top of previous releases: particularly in the area of motions and synchronizing settings between Visual Studio and VsVim.

I do want to take a second and again thank everyone who gave me feedback or filed issues throughout the entire development cycle. It really helps to have such great feedback to work against.

The usual caveats and expectations

This extension is being released by me, not by Microsoft. As such the support level for this extension is equivalent to the amount of free time I have to put into it.

Source for this release is available on the GitHub project site. It and the associated binaries are released under the MS-PL.

The update was actually released a few weeks ago but in my excitement I forgot to blog about it. ↩

DebuggerDisplay Attribute Best Practices

2011-03-18T00:00:00+00:00

The DebuggerDisplayAttribute is a powerful way to customize the way values are displayed at debug time. Instead of getting a simple type name display, interesting fields, properties or even custom strings can be surfaced to the user in useful combinations

[DebuggerDisplay("Student: {FirstName} {LastName}")]
public sealed class Student {
    public string FirstName { get; set; }
    public string LastName { get; set; }
}

The DebuggerDisplay attribute can customize the name, value and type columns in the debugger window. Each one can be customized using a string which can contain constant text or expressions to be be evaluated by the expression evaluator. The latter is designated by putting the expression inside {}’s (this greatly resembles String.Format)

This feature while very powerful and useful can also easily contribute negatively to the debugging experience when used improperly (mostly in the area of performance). After several years of working in this area and helping customers out with bugs I’ve come up with a few recommendations to help prevent this from happening ¹

Don’t use multiple functions or properties in the display string

Every time I see a DebuggerDisplay attribute like the following I cringe a little inside

[DebuggerDisplay("Student: {FirstName} {LastName} {Age} {Birthday} {Address}")]

Hands down the most expensive operation the expression evaluator does is evaluate a function. It dwarfs every other performance metric and can have a visible effect on stepping performance ².’? This is true for both functions and properties (as far as the debugger is concerned there is almost no difference between the two).

Every one of the expression holes above results in a property being evaluated. Each property must be evaluated individually and done so once for every instance of this type in every debugger display window. This set of evaluations is repeated on every single step. This can get very expensive if collections of this type end up getting displayed (imagine stepping with a couple thousand of these in the window!).

Please don’t read this and remove every property from DebuggerDisplay’s in your code. One property is very unlikely to cause a problem. Issues typically arise when many properties are used and collections of that type end getting displayed in the debugger windows.

Do use property / field names instead of language specific expressions

Evaluation holes in the string are not limited to just properties and function calls. They can handle pretty much any legal expression you can dream up. In the past this has lead to developers putting all manner of expressions into DebuggerDisplay attributes. The most common being the use of ternary expressions

[DebuggerDisplay("Count {IsEmpty ? 0 : Count}")]

While this works fine, please don’t do this!?? DebuggerDisplay attributes are evaluated not by the language in which they were defined but by the expression evaluator of the language in which they are being used. The above works great but only when viewed in a C# application. It fails miserably when viewed in other languages like VB.Net (and when F# has their own EE it will fail for that as well).

While there is no truly universal expression one which is supported by most languages is member names. Having an expression which is a simple property or field goes a long way to removing this problem.

Don’t evaluate expressions that throw exceptions

Earlier I mentioned that the most expensive action an expression evaluator performs is evaluating a function. The most expensive variant of evaluating a function are those which throw exceptions. Please don’t do this.

Don’t use mutating properties or functions

Usually this goes without saying but I’ve seen enough examples of this to warrant an entry. Don’t put expressions into DebuggerDisplay values which will mutate the underlying value. This will lead to only confusion.

Preferred Pattern

My personal preferred pattern for DebuggerDisplay attributes is to have the entire item be an expression: DebuggerDisplay. I then add a private instance property to my type named DebuggerDisplay and do all of my custom formatting in this property. Having the property be private is fine because nothing is private in the debugger.

[DebuggerDisplay("{DebuggerDisplay,nq}")]
public sealed class Student {
    public string FirstName { get; set; }
    public string LastName { get; set; }
    private string DebuggerDisplay {
        get { return string.Format("Student: {0} {1}", FirstName, LastName); }
    }
}

The ‘,nq’ suffix here just asks the expression evaluator to remove the quotes when displaying the final value (nq = no quotes).

I prefer this pattern because it’s only requires a single function to be evaluated, I can still have language specific expressions (which are nicely type checked by the compiler) and it doesn’t contribute to the public API of my type.

Note the word ‘I’. These are not any kind of official recommendations, just several I advocate to people using this feature. ↩
In one performance critical scenario for Visual Studio 2010 over 95% of it was spent evaluating the function! ↩

Vsvim Update Released Version 0 9 5

2011-03-14T00:00:00+00:00

I just released an update to VsVim for Visual Studio 2010. This is available on the extension manager in Visual Studio or can be downloaded directly at the following link.

Link: http://visualstudiogallery.msdn.microsoft.com/en-us/59ca71b3-a4a3-46ca-8fe1-0e90e3f79329

GitHub: http://github.com/jaredpar/VsVim

This update includes the following

Lots of undo / redo issues
Lots of caret positioning issues
Support for tabs
And of course many other bug fixes

I do want to take a second and again thank Martin Lemburg for the many detailed issues filed during the 0.9.4 release. It really helps to have such great feedback to work against.

Note: The latest release is 0.9.5.1. I released a patched version of 0.9.5 to deal with bad behavior in the ‘%’ motion which was breaking a lot of users.

Future Plans

I’m aiming for 0.9.6 to be a quick release cycle. The main focus of this release is getting macros and visual block operators working. I already have macro support working in the beta tree. I want to spend a few weeks working out the kinks and getting through other bug fixes and then I’ll release 0.9.6.

Macros represent the last major feature for the 1.0.0 release. So after 0.9.6 there will maybe be 1 more bug fix release before 1.0.0!

The usual caveats and expectations

This extension is being released by me, not by Microsoft. As such the support level for this extension is equivalent to the amount of free time I have to put into it.

Source for this release is available on the GitHub project site. It and the associated binaries are released under theMS-PL.

Yet Another Way to Defeat C++ const

2011-03-01T00:00:00+00:00

One of my favorite C++ features, and one I feel is terribly underutilized in many code bases, is the const mechanism. It’s a great mechanism for defining dual roles for the same object type: a mutable and (ideally) non-mutable one. But as useful as const is it’s also very easy to circumvent and I’m always interested in learning new ways to do so.

While navigating through a stackoverflow question yesterday I came across yet another way to defeat C++ const. In C++ it’s perfectly legal to call delete on a const value. This code path though transitions from a const * to a type to a non-const pointer by means of ‘this’in the destructor code. From there it’s possible to call any other mutable method in the type and bypass the original const without any nasty casts. For example

class Example {
public:
    ~Example() {
        MutableMethod();
    }

    void MutableMethod() {
        cout << "Example::MutableMethod()" << endl;
    }

    void ConstMethod() const {
        delete this;
    }
};

With this type I can easily defeat **const **restrictions by simply deleting it

const Example* pLocal = new Example();
pLocal->ConstMethod();

Granted this is not terribly useful when considering the **const **of the immediate type. It’s being deleted after all so messing around in it is unlikely to cause too much of a fuss. However it can be interesting when considering the access of nested members within the type which may live longer than the container being deleted.

Interesting Late Binding Scenario With Tostring

2011-01-24T00:00:00+00:00

Not to long ago I received an email from a customer who wanted to report a bug in the VB.Net debugger. They believed that there was a bug invoking ToString on Integer types in the immediate window and provided the following sample as evidence

i = 100
? i
100 {Integer}
    Integer: 100
? i.ToString("c02")
{"Conversion from string "c02" to type 'Integer' is not valid."}
    _HResult: -2147467262
    _message: "Conversion from string "c02" to type 'Integer' is not valid."
    Data: {System.Collections.ListDictionaryInternal}
    HelpLink: Nothing
    HResult: -2147467262
    InnerException: {"Input string was not in a correct format."}
    IsTransient: False
    Message: "Conversion from string "c02" to type 'Integer' is not valid."
    Source: "Microsoft.VisualBasic"
    StackTrace: "   at Microsoft.VisualBasic.CompilerServices.Conversions.ToInteger(String Value)"
    TargetSite: {Int32 ToInteger(System.String)}

The customer expected the method Integer.ToString(String) to be invoked and found the conversion to Integer to be a bug. Surprisingly to the user, and several people on the team, this behavior is ‘By Design’ ¹. To understand why we have to get a better picture of how exactly this is evaluated in the immediate window. There are two particular areas of importance here.

The static type of the variable i is Object not Integer. The first expression i = 1 declares a variable named i in the context of the debugger and assigns it the value 100. The ability to declare variables in the debugger predates type inference and uses Option Explicit Off semantics resulting in a type of Object for the variable
The debugger does not inherit project settings for Option Strict and instead evaluates all expressions with Option Strict Off.

These two combine together to mean that almost every expression evaluated on a variable declared in the immediate window will be done in a late bound fashion. It also means the above code sample is most accurately represented by the following real code.

Sub Main()
    Dim i As Object = 100
    Dim result = i.ToString("co2")
End Sub

Compiling and running that code will indeed cause the exact same exception as viewed in the immediate window. But why?

Remember earlier I said that almost every expression would be evaluated it a late bound fashion. The compiler will use late binding when it can’t find a suitable method to bind to statically and late binding is otherwise allowed. In this case the type of the variable is Object and hence Object.ToString() can be bound to statically and indeed that’s what happens in this case. Further in VB.Net it’s possible to call a method that has no parameters without parens: ex i.ToString is legal. This results in the (‘c02’) portion of the expression being interpreted as an indexer expression into the resulting string. Because Option Strict is off the compiler allows a silent narrowing conversion between String and Integer. The result of all of this is the code is actually evaluated as

Sub Main()
    Dim i As Object = 100
    Dim result = i.ToString()(CInt("co2"))
End Sub

I certainly found this interesting the first time I encountered it.

Please don’t confuse me saying an issue is ‘By Design’ with me thinking the behavior is ideal. It’s merely a statement that the behavior conforms to the specification at the time of this writing. ↩

Vsvim Update Released Version 0 9 4

2011-01-13T00:00:00+00:00

I just released an update to VsVim for Visual Studio 2010. This is available on the extension manager in Visual Studio or can be downloaded directly at the following link.

Link: http://visualstudiogallery.msdn.microsoft.com/en-us/59ca71b3-a4a3-46ca-8fe1-0e90e3f79329

GitHub: http://github.com/jaredpar/VsVim

This update includes the following

R# 5.1 Support
Snippet Support
Many fixes to key mapping
And of course many bug fixes

I do want to take a second and thank Martin Lemburg and rride for the many detailed issues they filed during the 0.9.3 release. It really helps to have such great feedback to work against.

Future Plans

I’m aiming for 0.9.5 to be a 3 week release cycle. It will expand the third party editor support to include DevExpress / CodeRush support and clean up many of the existing bugs. After that we’ll be pushing hard to get 1.0 out the door.

The usual caveats and expectations

This extension is being released by me, not by Microsoft. As such the support level for this extension is equivalent to the amount of free time I have to put into it.

Source for this release is available on the GitHub project site. It and the associated binaries are released under the MS-PL.

Why The Debugging Difference Between C And Vb Net Return Values

2011-01-12T00:00:00+00:00

A feature which seems to be getting more requests recently is support for seeing the return value of a function in the debugger without the need to assign it into a temporary. C++’s had this feature for some time but it’s been lacking in managed debugging scenarios. James Manning recently dedicated a couple of blog posts to the subject and noted that the feature appears to already partially exist for VB.Net

“With VB.NET, the function name shows up as an entry in ‘Locals’ and the debugger shows us the value it’s returning! The C# debugger has no such support, though - at the same ‘end of method’ breakpoint, only the parameter passed in is shown. So, clearly the CLR has enough support for the VB.NET debugger to support this feature, which would seem to be a pretty strong argument that the C# debugger certainly could implement this feature.”

Indeed C# could implement this feature but it’s not a CLR debugging feature that VB.Net is relying on but is rather an issue of VB6 legacy support. The VB6 language didn’t have a return statement. Instead values were returned by assigning the value to be returned to the name of the function. For example

Function IsEven(ByVal i As Integer)
    If i Mod 2 = 0 Then 
        IsEven = True 
    Else 
        IsEven = False 
    End If 
End Function 

While VB.Net added a Return statement it still supports this legacy syntax (and it allows the two to be mixed within a single function). The compiler models this by having a local of the same name of the function which is used to store the return value. Returns from the function are rewritten as assignments to this local and then a return of the same local. The debugger understands this hidden local and displays it during the debugging session. This gives VB.Net the appearance of supporting Return value display when in reality it’s just a positive side effect of legacy support.

Quick Note: In my experience when users ask for return value support in the debugger they typically want to see both

The return value of the current function they’re stepping through
The return value of functions which are stepped over (more heavily requested)

VB.Net supports only the first one (via the described method) and C# supports neither.

I do agree it would be really nice if both languages supported #2 (it’s an incredibly useful feature in C++). It is possible to do without CLR support but it involves the compiler generating a temporary for every function / property evaluated in a statement and lots of copying values around. This can have a non-trivial impact on program performance even when not debugging and hence I think is unlikely to be done. If #2 does come around it will likely be through the CLR debugger APIs providing access to the return values much in the way it provides access to the current exception.

Update

Starting with Visual Studio 2013 there will be return value debugging support for all managed languages in a uniform manner. It will be exposed in the same way that C++ exposes function return values. More details are available here

http://blogs.msdn.com/b/somasegar/archive/2013/06/26/visual-studio-2013-preview.aspx

Vsvim Update Released Version 0 9 3

2010-11-30T00:00:00+00:00

I just released an update to VsVim for Visual Studio 2010. This is available on the extension manager in Visual Studio or can be downloaded directly at the following link.

Link: http://visualstudiogallery.msdn.microsoft.com/en-us/59ca71b3-a4a3-46ca-8fe1-0e90e3f79329

GitHub: http://github.com/jaredpar/VsVim

This update includes the following

Full support for :substitute including the confirm option
Many bug fixes for the ‘.’ (repeat) command
Vim style regular expression support. Previous releases used BCL regexes
Support for the + (clipboard) register
And of course many bug fixes

Future Plans

The original plan for VsVim was to focus on getting a highly functional Vim emulator as the 1.0 release and then moving onto items like Resharper integration for 1.1. At our current pace this would put VsVim 1.0 out the door in just a few more weeks. But after receiving a lot of feedback from users it’s become clear that integration with Resharper is more important than rounding out the last few major Vim items. Hence we’ve decided to shift the focus of the next few releases to the following

0.9.4 ‘ Resharper support.
0.9.5 ‘ Finish remaining core Vim features (macros mainly at this point)
1.0 ‘ Mostly bug fixes on top of 0.9.5

We were really looking forward to putting the 1.0 stamp on VsVim but are equally as excited by the proposition of getting Resharper (and likely similar products like CodeRush) working. We hope to have a Beta release of the support by early next week for users to bash on. When the builds are available I’ll announce them on twitter and upload them to the github site.

The usual caveats and expectations

This extension is being released by me, not by Microsoft. As such the support level for this extension is equivalent to the amount of free time I have to put into it.

Source for this release is available on the GitHub project site. It and the associated binaries are released under the MS-PL.

Discriminated Unions In C

2010-11-18T00:00:00+00:00

Earlier this week I started writing a function which needed to represent three states in the return value, two of which had an associated value. In my mind I immediately came to the following solution

type BuildAction =
    | Reset
    | LinkOneWithNext of Statement
    | LinkManyWithNext of Statement seq

A discriminated union is perfectly suited for representing this type of scenario. Unfortunately for me I was coding in C++ and not F# so I shifted gears and started on a different solution. But I quickly grew frustrated with new solution and decided to backtrack. A discriminated union is just too ideal for this scenario so why not spend fifteen minutes and see how far I could get in defining a discriminated union structure in C++.

Of course a full discriminated union in the style of F# is not possible because C++ lacks language support for pattern matching. But I was willing to live with that and other limitations as long as I could get many of the other benefits I find with discriminated unions. In particular

Declarative Syntax
Typed value access without the need to cast
Allow mixing of flag only and flags with values
Customized naming of entries (no tuple style First, Second, etc ‘)
Instances are read only

After a bit of tinkering I came up with a solution (full source at end of post) which allows for the following declaration syntax

DECLARE_DISCRIMINATED_UNION(BuildAction)
    DISCRIMINATED_UNION_FLAG(BuildAction, Reset)
    DISCRIMINATED_UNION_VALUE(BuildAction, LinkOneWithNext, Statement)
    DISCRIMINATED_UNION_VALUE(BuildAction, LinkManyWithNext, vector<Statement>)
END_DISCRIMINATED_UNION()

Each declared entry in the union is provided the following methods.

A static factory method for creating values: CreateLinkOneWithNext
A method to test to see if an instance is the value type: IsLinkOneWithNext
A method to get the value associated with the associated value: GetLinkOneWithNext

    auto action = BuildAction::CreateLinkOneWithNext(statement);
    if (action.IsLinkOneWithNext()) {
        Method(action.GetLinkOneWithNext());
    }

It’s certainly far from perfect. But it did give me the tools to implement the solution in the way I inherently thought about it and freed me to spend my thinking time on other problems.

EDIT

A couple of people asked why I didn’t use boost::variant for this solution? Ideally this is the approach I would’ve taken. But for this particular scenario boost was unfortunately not an option (no weird management issues, just a boring production environment one).

Additionally the spirit of this post and experiment was just having a bit of fun and sharing the results (even though it does contain a few pieces of evil code).

DiscriminatedUnion.h

//----------------------------------------------------------------------------
//
// Discriminated Union in C++.
//
//----------------------------------------------------------------------------
#ifdef DECLARE_DISCRIMINATED_UNION
#undef DECLARE_DISCRIMINATED_UNION
#endif

#ifdef END_DISCRIMINATED_UNION
#undef END_DISCRIMINATED_UNION
#endif

#ifdef DISCRIMINATED_UNION_VALUE
#undef DISCRIMINATED_UNION_VALUE
#endif

#ifdef DISCRIMINATED_UNION_POINTER
#undef DISCRIMINATED_UNION_POINTER
#endif

#ifdef DISCRIMINATED_UNION_ALLOW_NONE
#undef DISCRIMINATED_UNION_ALLOW_NONE
#endif

#ifdef DISCRIMINATED_UNION_GET_KIND
#undef DISCRIMINATED_UNION_GET_KIND
#endif

#define DECLARE_DISCRIMINATED_UNION(name)       \
    struct name {                               \
    private:                                    \
        name() {}                               \
        unsigned __int32 m_kind;                \
    public:

#define DECLARE_DISCRIMINATED_UNION_WITH_NONE(name)         \
    struct name {                                           \
    private:                                                \
        unsigned __int32 m_kind;                            \
    public:                                                 \
        name() : m_kind(__LINE__) {}                        \
        bool IsNone() const {return m_kind == __LINE__;}

#define DISCRIMINATED_UNION_VALUE(unionName, entryName, entryType)                                          \
        static unionName Create##entryName(const entryType& value) {                                        \
            unionName unionValue;                                                                           \
            unionValue.m_kind = __LINE__;                                                                   \
            unionValue.m_##entryName = value;                                                               \
            return unionValue;  }                                                                           \
        bool Is##entryName() const { return m_kind == __LINE__;}                                            \
        const entryType& Get##entryName() const { ASSERT(m_kind == __LINE__); return m_##entryName; }       \
        entryType Get##entryName() { ASSERT(m_kind == __LINE__); return m_##entryName; }                    \
    private:                                                                                                \
        entryType m_##entryName;                                                                            \
    public:

#define DISCRIMINATED_UNION_POINTER(unionName, entryName, entryType)                                        \
        static unionName Create##entryName(entryType* value) {                                              \
            unionName unionValue;                                                                           \
            unionValue.m_kind = __LINE__;                                                                   \
            unionValue.m_##entryName = value;                                                               \
            return unionValue;  }                                                                           \
        bool Is##entryName() const { return m_kind == __LINE__;}                                            \
        entryType* Get##entryName() const { ASSERT(m_kind == __LINE__); return m_##entryName; }             \
        entryType* Get##entryName() { ASSERT(m_kind == __LINE__); return m_##entryName; }                   \
    private:                                                                                                \
        entryType* m_##entryName;                                                                           \
    public:

#define DISCRIMINATED_UNION_FLAG(unionName, entryName)                                                      \
        static unionName Create##entryName() {                                                              \
            unionName unionValue;                                                                           \
            unionValue.m_kind = __LINE__;                                                                   \
            return unionValue;  }                                                                           \
        bool Is##entryName() const { return m_kind == __LINE__;}                                            

#define DISCRIMINATED_UNION_GET_KIND() unsigned __int32 GetKind() const { return m_kind; }

#define END_DISCRIMINATED_UNION() };

Loading DLLs in the Debugger

2010-10-18T00:00:00+00:00

In a recent post I discussed the apparent flakiness of extension methods and the debugger being a result of whether or not the DLL containing the extension methods were loaded into the debugee process. Several users asked in the comment section why we didn’t fix the issue by just loading those DLL’s when an extension method was executed.

On the surface this seems like a reasonable request. After all the compiler knows what DLLs were involved in the compilation process and hence which extension methods were available from those DLLs. This information could be added to the PDB and used by the debugger to load the DLL when the extension method was requested. Right?

Unfortunately the answer is maybe which is leaning towards no. Lets ignore the performance and logistics around name lookup and instead focus on the debugger portion of the feature.

The first problem that comes up is which DLL should the debugger be loading? The PDB can tell us where the DLL’s involved in the compilation process were but that has no guaranteed relation to the current running program. Remember the machine the debugee process is running on is potentially (and likely) different than the one it was compiled on. The directory structure of the deployed application can (and will) be very different than the structure of the application when it was compiled. So any information the compiler could embed in the PDB about the path to the DLL is virtually useless.

Even if the paths matched perfectly how could the debugger be certain it was loading the DLL which was referenced in the compilation process over some DLL which just happened to have the same name’ True if the DLL was strongly named then a high degree of confidence could be established by doing a strong name check. But now we’re limiting the feature to extension methods coming from a strong named DLL. I don’t speak for all developers but the number of strong named DLL’s I’ve written is far outweighed by the non-strong named DLLs. With this sort of limitation what good would the feature be?

The next issue is how to reliably load the DLL into the process’ The debugger runs in a separate process and can’t simply call LoadLibrary to load the DLL. All communication with the debuggee process is done via the CLR and the debugging interfaces. They provide no API for directly loading a DLL. It can be achieved indirectly by executing a function in the process which has the side effect of loading the DLL (say Assembly::Load). This works but it is limited to situations where function evaluations are available. So scratch broken in native code, many optimized scenarios, etc ‘

The last issue is whether or not loading the DLL is worth the cost. Loading a DLL into an AppDomain is an operation that cannot be undone short of tearing down the AppDomain. Most applications have a single AppDomain so the effect of this operation lasts for the lifetime of the debugee process. Loading a DLL can produce unwanted side effects in processes which care about DLL load order or simply the set of DLLs in a process (aka any MEF application). In other words evaluating a simple extension method would irreparably change the state of the process! Probably not what the user intended.

Overall this adds up to a limited feature than potentially has a negative impact on user applications and is one the reasons it’s not used as a solution here.

Earlier though I said this feature was a ‘maybe leaning towards no’. The reason for the ‘maybe’ part is the feature is possible it’s just of questionable value. And it turns out that there are real scenarios where the positive benefits do outweigh the pitfalls. The primary example is the VB expression evaluator. It will force the Visual Basic runtime into the process when certain compiler helper functions are needed. These helper function are needed for a large set of VB expressions (late binding in particular) and hence has a high value to users. The DLL is also strongly named and available in the GAC so there is little fear of loading the wrong DLL. Nor is the introduction of the runtime into the process considered dangerous as it’s the default for VB projects.

Vsvim Update Released Version 0 9 2

2010-10-15T00:00:00+00:00

I just released an update to VsVim for Visual Studio 2010. This is available on the extension manager in Visual Studio or can be downloaded directly at the following link.

Link: http://visualstudiogallery.msdn.microsoft.com/en-us/59ca71b3-a4a3-46ca-8fe1-0e90e3f79329

GitHub: http://github.com/jaredpar/VsVim

This is fairly major release from the last announced (0.8.2) which includes the following changes

Essentially a rewrite of Visual Mode
New motions f, F, t, T, e, [, [[, ], ]], (, ), {, }
Visual Studio integration improvements
Memory leak fixes
Fixes for international keyboard issues
And of course many bug fixes

I do want to take a second and thank two VsVim users, pedrosal and tuncbah, for reporting several international keyboard issues, helping me to understand the problem and testing out my fixes. They were particularly nasty issues that caused me a bit of trouble.

Future Plans

We’re quickly approaching a 1.0.0 release. My initial goal for 1.0.0 was to have a largely functional Vim emulation layer on top of Visual Studio. With the following features currently in progress I think VsVim will be well within that goal

Vim style regex support - right now all regex’s are done with vanilla BCL regex’s
Clipboard support
Improvements to the . operator

This work is well under way and I hope to complete it within the next month or so. Releasing 1.0 will be a huge milestone for us but will by no means be the end of the work.

The usual caveats and expectations

This extension is being released by me, not by Microsoft. As such the support level for this extension is equivalent to the amount of free time I have to put into it.

Source for this release is available on the GitHub project site. It and the associated binaries are released under the MS-PL.

Improving the Display of seq in the Debugger

2010-09-10T00:00:00+00:00

F#’s seq expressions are a frustrating item to inspect at debug time. A seq value is a collection and when users inspect such a value at debug time they want to see the contents of the collection. Instead they are often presented with a view resembling the following

The reason this happens is a consequence of how the debugger works. When a value is expanded it essentially enumerates the members and display the values as child nodes. This is great for most types of objects where the members hold the mots interesting data. For collections though it’s often far more interesting to see the elements in the collection.

Most collection types don’t have this issue because they are either custom handled by the expression evaluator, arrays for example, or use a debugger type proxy helper class (Dictionary<TKey,TValue>, List, etc ').'? One class of collections that do not display well are C# iterators and F# seq expressions. Both languages translate these into generated types which expose IEnumerable. The type contains no custom type proxy helper classes and expression evaluators don't custom handle them. Users cannot fix the problem either by adding a custom DebuggerTypeProxy because they are generated types and hence not directly accessible to the user.

The languages team realized this was a problem and added a debugger feature in Visual Studio 2008 to help: the ‘Results View’ node. When an expression appears in the locals or watch window which is typed to IEnumerable and has no associated DebuggerTypeProxy the expression evaluator will add a special child named Results View. When this is expanded it will enumerate the IEnumerable and display the values. Here is the equivalent C# sequence displayed in 2008 and beyond.

This display is much better because it’s actually showing me the elements in the IEnumerable!!!

Astute readers may be wondering at this point why this doesn’t work for F# out of the box. After all F#’s seq maps to IEnumerable and the F# debugging experience uses the C# expression evaluator. So there should be no difference in the experience here.All of this is true, there's just one small problem. The C# (and VB) expression evaluators don't actually do the work of enumerating the IEnumerable. Instead they rely on a helper class defined in System.Core.dll. If this DLL is not loaded then the Results View node can't be created. The C# and VB.Net project system work to have System.Core, and other important DLL's, loaded at the start of a debugging session. The F# project system does not do this and hence misses out on this experience by default.

Developers can get this experience in F# by forcing System.Core.dll into the debugee process. My favorite trick is to add the following line at the start of my F# applications.

#if DEBUG
System.Linq.Enumerable.Count([]) |> ignore
#endif

This forces System.Core.dll into the process in debug builds which results in a much nicer display for any seq expressions I have.

When I originally posted this solution to a question on StackOverflow, Tim Robinson pointed out that it’s possible to force System.Core.dll into the debugee process without adding extra debug code to the process. Simply use the Assembly.Load API to do it by evaluating the following expression in the watch window.

System.Reflection.Assembly.Load(“System.Core, Version=3.5.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089, processorArchitecture=MSIL”

I still prefer my solution but this is a great trick to have if developers are in a situation where they can’t edit the code. Such as debugging in a production environment.

Convirting System.Func to FSharpFunc

2010-07-27T00:00:00+00:00

Interop of delegate style types between F# and other .Net languages is a pain point that results from a fundamental difference in how delegates are represented in the F# language. Typical .Net languages like C# see a delegate as taking 0 to N parameters and potentially returning a value. F# represents all exposed delegates as taking and returning a single value via the FSharpFunc<T,TResult> type. Multiple parameters are expressed by having the TResult type be yet another FSharpFunc<T,TResult>.

This creates a host of problems calling exposed F# delegate / function types

Can’t use C# lambda expressions which take more than one parameter
System.Func<> expressions of equivalent logical shape can’t be converted
Method group conversions don’t work

The only step is to introduce a conversion step when calling into F# code. F# does provide a helper method in F# in the FSharpFunc.FromConverter method. But once again it’s only useful for delegates of one parameter, anything higher requires a more in depth conversion. For example here is the C# code to convert a 2 parameter delegate into the equivalent F# type.

public static FSharpFunc<T1, FSharpFunc<T2,TResult>> Create<T1, T2, TResult>(Func<T1,T2,TResult> func)
{
    Converter<T1, FSharpFunc<T2, TResult>> conv = value1 =>
        {
            return Create<T2,TResult>(value2 => func(value1, value2));
        };
    return FSharpFunc<T1, FSharpFunc<T2, TResult>>.FromConverter(conv);
}

Not very pretty or intuitive because the code needs to recreate the nested style of F# func’s. This gets even more tedious and error prone as it gets past 2 parameters.

The simplest solution, as is true with most F# interop scenarios, is to leverage F# itself to define the interop / conversion layer. It already naturally creates the proper nesting structure so why spend type redoing the logic in C#. The logic can then be exposed as a set of factory and extension methods to make it easily usable from C#.

    [<Extension>]
    type public FSharpFuncUtil = 

        [<Extension>] 
        static member ToFSharpFunc<'a,'b> (func:System.Converter<'a,'b>) = fun x -> func.Invoke(x)
    
        [<Extension>] 
        static member ToFSharpFunc<'a,'b> (func:System.Func<'a,'b>) = fun x -> func.Invoke(x)
    
        [<Extension>] 
        static member ToFSharpFunc<'a,'b,'c> (func:System.Func<'a,'b,'c>) = fun x y -> func.Invoke(x,y)
    
        [<Extension>] 
        static member ToFSharpFunc<'a,'b,'c,'d> (func:System.Func<'a,'b,'c,'d>) = fun x y z -> func.Invoke(x,y,z)
    
        static member Create<'a,'b> (func:System.Func<'a,'b>) = FSharpFuncUtil.ToFSharpFunc func
    
        static member Create<'a,'b,'c> (func:System.Func<'a,'b,'c>) = FSharpFuncUtil.ToFSharpFunc func
    
        static member Create<'a,'b,'c,'d> (func:System.Func<'a,'b,'c,'d>) = FSharpFuncUtil.ToFSharpFunc func

Now I?? can convert instances of System.Func<> to the F# equivalent by simply calling .ToFSharpFunc().

var cmd = Command.NewSimpleCommand(
    name,
    flagsRaw,
    func.ToFSharpFunc());

Much better.

Advanced Multitargeting In Vb Net

2010-07-23T00:00:00+00:00

Multi-targeting is a feature introduced in Visual Studio 2008 which allows developers to use new versions of Visual Studio to target earlier versions of the .Net platform. It allowed users to target both the new 3.5 and 3.0 and the previous 2.0 profile with the same IDE. Visual Studio 2010 continues this trend by adding support for CLR 4.0 and even allows for further sub-targeting through several framework profiles.

At a compiler level this works by providing binary compatibility with previous versions of the CLR (or producing an error when it’s not possible). It doesn’t warn developers about using new language constructs on older frameworks so long as they function in the target framework.
Take the following code sample as an example. It is taking advantage of implicit line continuations which is a feature introduced in VB.Net 10 and not available in Visual Studio 2008. Since it’s just syntactic sugar it compiles just fine in an application targeting 4.0 or any previous version of the framework.

Dim query =
    From it In col
    Where it Mod 2 = 0
    Select it * 3

In a limited set of scenarios though developers want to use Visual Studio 2010 to develop code which will also be compiled by a previous version of the compiler. Hence they want warnings in Visual Studio when new language constructs are used.

The 10.0 version of vbc.exe introduced a new switch named langversion to provide just that. It allows developers to specify a version of the language to target. The compiler will then issue an error when a language construct which was introduced in a later version is used. This switch is not available from the IDE though and must be manually specified in the project file. To do so edit the .vbproj file and add the following element under the main ItemGroup element.

<PropertyGroup>
    <LangVersion>9</LangVersion>
</PropertyGroup>

Once the project is reloaded errors will start appearing for new language constructs. Notice how our earlier sample now produces the errors for the usages of the new implicit line continuation construct.

Extension Methods And The Debugger

2010-07-22T00:00:00+00:00

One source of confusion I find myself clearing up a lot is the use of evaluating extension methods in the debugger windows. Users report evaluation as working sometimes but not others for the exact same piece of code. Such flaky behavior can only be the result of a poorly implemented feature or subtle user error. Right’

Unfortunately no. In this case the behavior described is very possible and ‘By Design’¹. It’s an unfortunate fallout from how the way the debugger works.

Quick review. Expression evaluators strive to have evaluation parity with the compiler. So if expression expr _is valid at the place the debugger is stopped, _expr should also be a valid expression in the immediate, watch, etc … windows. This holds true for extension methods. For example

using System;
using System.Collections.Generic;
using System.Linq;
using System.Diagnostics;

namespace ConsoleApplication1 {
    class ExtensionMethodExample {
        public static void Example() {
            var col = new List<int>();
            col.Add(42);
            Debugger.Break();
        }
    }
}

At the Debugger.Break line expressions like col.First() are valid and legal (assuming System.Core is referenced). Hence they should also be available in the debugger windows. But with extension methods developers will occasionally see the following.

Clearly it failed to evaluate but is legal in code so users often interpret this as a bug (and I can’t blame them, the behavior is odd).

Expression evaluators host the compiler in order to evaluate expressions. In order to semantically interpret an expression compilers need symbols to bind to. In a debugging session symbols are acquired by reading the metadata from the DLLs loaded into the debugee process. So the evaluation essentially occurs by referencing the set of DLL’s loaded into the debugee process.

For most expressions this poses no problem. In order to even have a given value to run an expression off of it’s DLL must be loaded and hence symbols for the type of the value and it’s members are available. Extension methods are quite different though in that the target method can, and often does, live in a separate DLL. So unlike normal members, simply having a value in the debugger does not necessitate symbols for it’s extension method are loaded into the process. When they are not binding fails.

This is the case for the above sample. The symbols for the value ‘col’ are in mscorlib while the ‘First’ extension method are in System.Core.dll. If System.Core.dll is not loaded into the process then it’s symbols are not available and attempts to bind to the LINQ extension methods will fail.

This is what makes the behavior appear to be flaky. The ability to call an extension method is directly related to whether or not it’s DLL is currently loaded in the debugee process.’? When there is a disconnect between DLL’s available at compile time and loaded in the process there becomes a gap in what can be evaluated.’? DLL’s are loaded on demand and if no extension method, or other type, in the DLL has been used up until the current point in the process it will not be loaded and hence not available.

What complicates this discussion even further is a side effect of the hosting process in Visual Studio is that it hides this problem for certain DLLs (primarily System.Core). One of the features of the hosting process is that it preloads a set of DLL’s into the debugee process including System.Core.dll. As a result LINQ extension methods are readily available in most projects. For a normal console application the above won’t ever fail with F5 unless you specifically disable the hosting process in the debug tab of the project properties page.

This further adds to the perception of extension methods in the debugger are flaky since LINQ works but user defined extension methods fail. It creates additional confusion because the hosting process does not work for all project types (devices, certain types of web projects, etc ‘) and does not come into play in an attach scenarios.

failing but such is life.

I do hate using the ‘By Design’ tag to describe a feature as successfully ↩

Disable Copy Construction When The Type Is Not Copy Safe

2010-07-12T00:00:00+00:00

A couple of days ago I finished coding up a feature in our C++ code base, hit F5 and was met with a nasty memory corruption debugger dialog. After about an hour of investigation it appeared one of my types was living past the lifetime of it’s owning heap. I decided the next step was to debug on the heap functions to see where I went wrong. I opened up the file for the heap and almost immediately new what I had done wrong.

Here’s a trimmed down version of the heap type I was looking at.

    class SpecialHeap {
    public:
        SpecialHeap() { 
            m_pBlock = ::VirtualAlloc(...);
            // Memory management magic
        }

        ~SpecialHeap() {
            VirtualFree(m_pBlock, ...);
        }
    
        void* Allocate(sizet_ bytes) { }
    private:
        void* m_pBlock;
    }

Anyone else immediately guess what I did wrong¹?

What’s relevant is the code that’s not on the screen. Namely the default copy constructor generated by the C++ compiler. Types in C++ which don’t explicitly define a copy constructor are given a default copy constructor by the compiler. This is great for several classes of types in C++ as it allows for handy value copying with no extra user code.

One type it’s terrible for though are those which manage resources that are not meant to be shared. Types like our SpecialHeap for example. Any instance of SpecialHeap which is created via a copy constructor is a time bomb. A copy means there is an original and hence 2 SpecialHeap instances running around which refer to the same m_pBlock value. After the first destructor runs the other instance is left holding a garbage pointer.

After taking one look at this I knew I almost certainly left a & off of a SpecialHeap parameter which I intended to pass by reference. Instead it created a copy and when the function returned it destroyed the heap memory I expected to be alive. A quick scan of my change verified this was indeed the bug.

What’s really frustrating is that this bug is 100% preventable. Types can opt out of copy construction by simply declaring a user defined copy constructor with no implementation. ²

    private:
        // Disable value copying
        SpecialHeap(const SpecialHeap&);
        SpecialHeap& operator=(const SpecialHeap&);

This code just turned my runtime bug into a compilation error. What’s very frustrating about this issue is that it took < 1 minute to type and would have saved me an hour of debugging had the original author taken the time to add it. This type is not copy safe and allowing it to have a copy constructor is simply a bug.

This pattern is not special to this bug. It should be done for every single type you ever define in C++ that you do not explicitly intend to be copied. Getting in the habit of adding the above code **will **save you and others hours of debugging at some point down the road.

The title of the blog post is a big give away ↩
Making it private has the added bonus of being a binding error in almost all cases vs. a linker error ↩

Vsvim Update Released Version 0 8 2

2010-07-09T00:00:00+00:00

I just released an update to VsVim for Visual Studio 2010. This is available on the extension manager in Visual Studio or can be downloaded directly at the following link.

Link: http://visualstudiogallery.msdn.microsoft.com/en-us/59ca71b3-a4a3-46ca-8fe1-0e90e3f79329

GitHub: http://github.com/jaredpar/VsVim

This is a bug fix release and contains no large features.

Notable Bug Fixes

Both normal mode append commands, ‘a’ and ‘A’, now correctly position the cursor after the last character of the line in all cases.
Several issues around the display of the : in command mode
Motions e/E now correctly move past the end of the word when used as a movement
The dw edit lead to exceptions when deleting some times of leading whitespace
Tooltips no longer interfere with normal mode commands
Several subtle motion issues

Future Plans

We are primarily working on 0.9.0 features at this point which can be seen here. This release is continuing to fill in the gaps users have reported. Depending on how long 0.9.0 takes there may be a 0.8.3 release but right now it looks like 0.9.0 will be the next version.

The usual caveats and expectations

This extension is being released by me, not by Microsoft. As such the support level for this extension is equivalent to the amount of free time I have to put into it.

Source for this release is available on the GitHub project site. It and the associated binaries are released under the MS-PL.

Do Not Throw A Nullreferenceexception When Validing This In An Extension Method

2010-06-28T00:00:00+00:00

One pattern I’ve started running into is developers explicitly throwing a NullReferenceException when validating the ‘this’ parameter of an extension method.

public static void ForEach<T>(this IEnumerable<T> enumerable, Action<T> action)
{
    if (null == enumerable)
    {
        throw new NullReferenceException();
    }
    // rest omitted 
}

The desired behavior here is to make extension methods look even more like instance methods by adding similar exception semantics. So now a call to col.ForEach(..) throws a NullReferenceException if col is null just like it would if ForEach were a real method.

However this is incorrect and should be avoided.

A null reference has not occurred in this example. Having an exception targeted to a very specific event being raised when that event did not occur is simply incorrect.
NullReferenceException is a runtime exception and should only be raised by the runtime. In several cases the runtime attaches special semantics to an exception it throws that changes the way it is handled (StackOverflowExcepion for example). Throwing runtime exceptions in user code means certain catch handlers can now potentially execute it at least 2 ways. This only serves to confuse developers?? ¹
Extension methods can be, and often are, still called just like a plain old static method and must play by those rules. The .Net framework guidelines are very clear on how this case should be handled.
Extension methods can validly be called on a null ‘this’ value and it doesn’t represent an intrinsic error as it does for a normal instance method ²

For me #1 and #3 are the most compelling points. Violating either of these goes against established practices and semantic and leads to incorrect behavior in programs.

The correct pattern here is that for a normal static method: use an ArgumentNullException.

public static void ForEach<T>(this IEnumerable<T> enumerable, Action<T> action)
{
    if (null == enumerable)
    {
        throw new ArgumentNullException();
    }
    // rest ommitted 
}

I very much wish explicitly throwing runtime exceptions produced unverifiable code ↩
Whether or not this is a good idea is a different question ↩

Avoiding Automation Bugs When Implementing IOleCommandTarget

2010-06-10T00:00:00+00:00

Shortly after Visual Studio 2010 shipped I wanted to experiment with the new VSIX format for traditional Package extensions. I fired up my copy of Visual Studio, ran through the new package project wizard. But instead of a nice shiny new project I was greeted with a project load error dialog. After a bit of investigation I found the generated project file was corrupt. The majority of the template code was not replaced.

A bit disturbed that we shipped such a bug I immediately fired an email off to the appropriate team and inquired about the situation. I got back a very quick psychic debugging response.

Do you have any extensions installed on the machine?

After a quick check I verified the only extension on my machine was my very own VsVim project. I naturally assumed this couldn’t be any fault of VsVim because it doesn’t participate?? in project creation and no other project type was affected. Imagine my surprise then that after uninstalling VsVim everything started working as expected. After a few more verification runs, and a bit of self shame, I realized it indeed was my problem and got out the debugger.

It turns out the package wizard does project file substitution a bit differently than the project project templates. It uses the DTE object model to do the replacements. DTE is different than the other VSIP interfaces in that it is intended to participate in automation (aka Macros).’? Operations tend to avoid manipulating the buffer directly but instead raise a command which is later handled by another component in the editor. Using the command system allows DTE to participate in automation.

For example operations like TextSelection.Delete don’t directly delete text from the buffer. Instead it just raises the command which represents the user hitting the delete key which is eventually handled by the editor and deletes the selection. These commands are processed through the IOleCommandTarget chain for a given IVsTextView.

Once I found out this it became immediately apparent what was happening. The package wizard is effectively sending keystrokes to a buffer with the intend of performing edits. VsVim uses IOleCommandTarget to intercept key strokes and was happily treating them as Vim commands.

The implications are even further reaching than just the package wizard. Macros primarily operate on the DTE object model hence I was also breaking them (very badly indeed).

Luckily there is a very simply fix to the problem. Visual Studio provides a nice helper method which allows you to determine if you are currently in the middle of automation. Adding a simple check for this at my IOleCommandTarget entry points cleared up the issues nicely.

if (VsShellUtilities.IsInAutomationFunction(_serviceProvider))
{
    return false;
}

In general any component in the IOleCommandTarget chain should be making the same check on both QueryStatus and Exec. Unless your component is specifically designed for macros and automations handling a command during automation will lead to hard to track down issues.

Newlines In The Immediate Window

2010-06-08T00:00:00+00:00

A question came up recently on stack overflow concerning the display of newlines in the immediate window.’? The author noted that any .ToString method which contained a newline printed incorrectly when evaluated in the immediate window. For example given the following ToString implementation

public override string ToString() {
    return "Hello" + Environment.NewLine + "World";
}

The immediate window would display the following. Noticed how the newline was printed as the actual escape sequence instead of a physical newline.

This behavior is not limited to ToString calls but will occur for any place an expression evaluates to a raw string.’? This behavior in the immediate window case is a fallout from the architecture of the debugger components.

The current level of fidelity in the debugger APIs prevent the expression evaluator from knowing which window it is providing data for. It is simply asked to evaluate expressions and provide data with very little in the way of context. In some ways this is a good thing because it trims down the number of scenarios to test because it limits the number of ways it is produced. The downside of course is that data provided must be able to work in every window and hence a bit more generic. Often times this leads to trade offs in how data is formatted.

Newlines are one of the places where a trade off was made. The immediate window is unique in that it’s the only window in the debugger where the default display contains multiple lines. Every other window (watch, locals, autos, etc ‘) contains a single line for display. Attempting to display a multiline string in a single line results in either having only the first or last line be visible. Hence the expression evaluators chose to escape the newlines to make more of the string visible in the majority scenario ¹

In the case this happens there is a built-in debugger visualizer which allows for the string to be displayed without any escaping. Simple click on the magnifying glass and select the Text Visualizer option. This pops up a modal dialog displaying the string in an unaltered form

This escaping does not always happen. Essentially only when the data is specifically typed to a System.String ↩

My Next Adventure

2010-06-07T00:00:00+00:00

After 4+ years on the languages team it’s time for me to move onto the next adventure. I joined the languages team in the winter of 2006. Since then I’ve worked on pretty much every part of the language experience including the compiler, debugger and IDE, shipped 2 releases of Visual Studio, several independent projects and fixed the occasional bug.

I will be joining an operating systems incubation project inside of Microsoft. It’s an exciting project which is exploring the world of multi-core programming and other aspects of the stack that I am deeply interested in.

The downside of working on a team for 4+ years though is you tend to get rather attached to the people and product. Visual Studio and in particular the languages team is full of a lot of great people who’ve had a lasting professional and personal impact on me.’? It was a great experience and I will miss it all.

This of course won’t stop me from continuing to blog on Visual Studio, languages, the BCL, .Net or really anything else that comes to mind. Nor will it prevent my continued work on Visual Studio extensions such as VsVim. I’ll just be doing all of it from a different building.

Using Lambdas To Create Generic Factories

2010-06-04T00:00:00+00:00

One item I find to be limiting in C# is the new generic constraint. The syntax construct specifies that the type backing a given generic parameter contains a parameter less constructor.’? It allows methods to create instances of generic parameters in a type safe manner.

public static void Example<T>() 
    where T : new()
{
    var value = new T();
    Process(value);
}

I find though that I very rarely want this capability. The object can’t be provided any initial state since you can’t give it any values to the constructor. It’s really only useful if you want to create and then mutate a given object.

Typically I prefer to deal with immutable data or at least types which build upon other information. Hence my types tend to have constructors which take at least one piece of information. In this case the new constraint is of no value because it can’t be used to describe arbitrary constructor signatures but is limited to only a parameter less constructor.

public static void Example<T>()
    where T : new(string) // Not possible!!!
{
    var value = new T("some data");

}

Originally when I faced this situation I ran to the factory pattern. I created a nice IFactory interface which had a Create method taking a string, and modified the methods to take this type instead.

public static void Example<T>(IFactory<T> factory) {
    var value = factory.Create("some data");
    ...
}

This works and there is nothing wrong with it. Except of course it’s an extremely verbose solution. Every time I want to use the method with a new type I have to create a new type which implements IFactory<MyNewType> just to call this method.’? This is very tiresome and gets frustrating very fast with broad object hierarchies.

Fortunately there is a much lighter weight solution to this problem: lambda expressions. All we need in this instance is a method which given one or more pieces of input returns a new instance of T. How this is implemented is of no concern to the method. This can easily be done via a delegate.

public static void Example<T>(Func<string, T> createInstance) {
    var value = createInstance("some data");
    ...
}

Now the caller can provide the contract with a simple light weight lambda expression

Example(data => new Widget(data));

Why Is Linq Absent From Debugger Windows Part 2

2010-06-02T00:00:00+00:00

Some readers may remember an article I published almost half a year ago about LINQ being absent from the debugger windows. That post explored the initial design of the feature, it’s limitations and ultimately why it was absent but promised a future article on a slightly different approach. It’s quite late but I’ve finally had some time to write the second part of this article ¹

No, I didn’t forget about the article or really get too lazy. The pace of Dev10 really picked up shortly after publishing that article and the follow up was put on the back burner. This week I finally find myself with a bit of free time and decided to follow up on my promise.

First a quick refresher. Features in the Expression Evaluator are started on the basis of having complete parity with the feature as it exists in the language and then paired down based on cost. Having complete parity with LINQ is prohibitably costly due to it being much more of an ENC expression instead of an inspection one (the latter being the primary purpose of an expression evaluator). Now comes the compromise phase.

As stated in the previous article the most expensive portion of this design is adding ENC support to the Expression Evaluator. ENC is necessary for true parity because closures in C# and VB.Net are mutable and requiring altering program state to implement. The standard example is evaluating the following expression in the EE which modifies the value of a local variable.

((Action)(() => { local1 = 42; }))();

Properly implementing this requires program mutation not just inspection.

Now comes the compromise. How often do users really want to do this’ In my observations the answer is rarely if ever. When I talk with customers about LINQ in the debugger windows almost every single answer comes down to allowing filtering / where expressions. These are inherently, but not strictly, non- mutating operations.

list.Where(x => x.Name == "Jared");

Cutting mutation is a huge advantage because we can forget about the impact into existing closures and alteration of program structure. Instead we can generate new closures which copy the initial state of the values and work from there. This removes all of the state tracking problems examined in the previous article.

If we cut mutation, and hence ENC, we can move to a different approach for generating code. Instead of generating all of the types and methods in the current assembly with ENC, create a generated assembly in the target process which contains the new types and methods. This can be done through existing technologies like Reflection.Emit.

So How does that change the cost of the feature’ If we look at the major feature list discussed in the previous article we’d only be left with the following

A metadata generation service to support the backing for closures and lambda expressions
Converting expressions typed in the EE to IL

Lets take a deeper look at this given our new proposed architecture.

Metadata Generation Service

The debugger and expression evaluator live in a different process than the debugee. Any generated lambda expression needs to be done in the debugee process. Yet all of the information needed to create the lambda exists in the debugger process. Implementing this feature requires that all of this data to be transferred between the processes. In effect we’d have to serialize both the metadata and IL of the trees and send them across the process boundary.

Creating a DLL to host the service, defining the interfaces and calling them from the EE all have costs. Yet many of these are done on every release and the cost is very much understood. Due to the regularity with which we do this we can cost this part with high confidence.

What’s expensive here is defining the format for the data transfer. Today we don’t directly generate IL in our compilers but instead use services exposed by the CLR to do so. We would need to define a new serialization format for transferring our representation of trees to the debugee process, deserializing this in the debugee and converting it to metadata and IL. This is actually very costly when you consider all of the details

Needs to work in both a 32 and 64 bit environment
The sheer amount of data as the number of nodes in our trees is quite large
Data is transferred from a native process to a managed one so the structure for serializing data gets defined twice
Versioning:
- This service must support both the current and new versions of the EE
- The serialization format must be able to handle the changes we make to our nodes over the course of several releases
Performance

Of all of these ‘versioning’ is the biggest cost. All to often we don’t consider versioning when designing EE features and it comes back to haunt us in the long term.’? To implement such a large feature and not consider the versioning aspects would be a huge mistake.

The lack of previous work in this area and the overall size produces a feature with a high cost which has low confidence. Not what a manager wants to hear.

Converting Expressions in the EE to IL

The expression evaluator does not actually compile expressions down into IL. Instead it compiles it down to a low level semantic tree which is then directly interpreted by the expression evaluator.

This approach grants a lot of flexibility to the EE and allows us to evaluate expressions that are not necessarily valid at the current place in the program. For example the ability to directly evaluate object id expressions. This flexibility hurts us here because it creates a subset of expressions which are not, or at least very difficult to, translate to IL. How would an object id expression for instance be expressed in IL?

The problem is not just limited to object id expressions but also include a host of other items allowed in the EE. There are too many to list in this article but suffice it to say this is not a trivial problem to solve ². Any cost for this area would have at best a medium level of confidence.

The next problem with this approach is that our infrastructure which generates IL is heavily geared towards doing so for EXEs and DLLs. It has been so for every release of our code base and contains a lot of code very specific to this process. Converting this to be more general purpose is very costly at this point. It would entail almost a complete rewrite of those components.

Once again this cost in itself is not prohibitive but does add up. Really it amounts to a known type of refactoring with medium to high confidence.

Wait there’s more: Don’t forget testing

There are two costs to consider for the testing of this new feature. The first is the straight forward costs you get with any new features. This is a pretty standard and well understood process. The language side of the feature is not too difficult to test. The cost really starts adding up though when you consider all of the other parts which can go wrong: loading DLL’s into the process, 32 / 64 bit issues, making sure all instructions serialize, versioning, etc ‘

The second more hidden cost though is the impact of this new feature on existing ones. Consider again that today all expressions typed into the EE are interpreted. However with our current design if the expression was inside of a lambda expression it would be executed not interpreted. This is a completely different process and would require a completely different set of tests to verify it’s functionality.

In effect this would double the cost for QA for all existing features as they’d need to be tested inside a lambda expression. This is an enormous cost and one that cannot be ignored.

In summary

Here is a summary of the larger cost items

Define a version friendly serialization format for transferring metadata and IL across the native and managed boundaries
Implement both the native serializer and managed deserializer for this format
Implementing a metadata generation service
All of the work around getting this DLL into the target process
Finding solutions for generating IL for all expressions which don’t directly map to IL
Refactoring our existing compiler infrastructure to be friendly to generating IL for the EE
QA costs for testing this feature
QA work to double test every type of expression (inside and outside of a lambda)

This doesn’t take into account a lot of the smaller items I’ve ignored ³ or the items we’d only find once we started implementing. This is after all a very large feature and those always have hidden costs that aren’t found until implementation hits a certain point.

I’m very hesitant to put a strict time estimate on this feature (it would almost certainly be wrong given the hidden costs and the lack of confidence in estimating several areas). To put it in a bit of context though, I would estimate it as large and likely larger than any other feature I’ve worked on since I joined Microsoft.

Once again I’m not writing this blog post to justify why we won’t ever implement LINQ debugging. Instead I write this to justify why we haven’t done it up till this point. This is a feature which has clear customer value and a fair bit of demand and I want to help customers understand our decision making process in this area.

I’m still very hopeful this feature will make it into the product at a future release. Perhaps we’ll find a cheaper route we’re not currently considering. Or maybe a future feature we need will offset some of the costs here.

This gap is certainly a personal record and one I hope to never beat ↩
One day I may write about these items because pretty much all exist to support a richer user experience. ↩
Anonymous types, transparent identifiers and VB generated delegates for starters ↩

Vsvim Update Released Version 0 8 1

2010-06-01T00:00:00+00:00

I just released an update to VsVim for Visual Studio 2010. This is available on the extension manager in Visual Studio or can be downloaded directly at the following link.

Link: <http://visualstudiogallery.msdn.microsoft.com/en-us/59ca71b3-a4a3-46ca- 8fe1-0e90e3f79329>

GitHub: http://github.com/jaredpar/VsVim

New Features

I forgot to add an entry when 0.8.0 was released so this is the main set changes for both 0.8.0 and 0.8.1.

Motions: G, gg, H, M, L, f, t, F, T
Options: startofline, virtualedit, visualbell, hlsearh, ignorecase, smartcase
Visual Mode ‘ put, <, >, ~, all motions from normal mode
Normal Mode ‘ D, ~, N, Ctrl-F, Ctrl-B,
Command Mode ‘ qa

Notable Bug Fixes

VsVim will now look for a .vsvimrc file before .vimrc. VsVim still does not support the full set of commands and certain combinations in users .vimrc caused issues.
Closing a file while in Insert Mode will no longer cause an exception
Users can now map F1 properly in key remappings. Useful in laptops where F1 is in the place of Escape

Future Plans

I will likely be releasing a 0.8.2 version in very early June when I return from vacation to clean up a few items that just didn’t make it into 0.8.1.

The current set of planned features for 0.9.0 can be seen here.?? Mainly this is filling out the remaining gaps users have reported. I’ve heard a few requests now for R# support in VsVim and that will be coming. I do not know yet if that will go into 0.9.0 or a later release. Will likely depend on how much feedback I get on this.

Special Thanks

I wanted to take a second and thank all of my users who took the time to file issues and work with me to get a solid repro so I could fix the bug. VsVim is currently a very community driven product and I would like to see it continue to be.

Additionally thanks to JasonMal a coworker of mine who took the F# plunge and started helping me fix issues. The help is most appreciated.

The usual caveats and expectations

This extension is being released by me, not by Microsoft. As such the support level for this extension is equivalent to the amount of free time I have to put into it.

Source for this release is available on the GitHub project site. It and the associated binaries are released under the MS-PL.

Metablogging Changes To My Blog

2010-06-01T00:00:00+00:00

Recently the MSDN blog architecture got a long overdue update to a new infrastructure. Most of these changes are very welcome but a few are causing a bit of issues for me and readers

My blog URL changed to - http://blogs.msdn.com/b/jaredpar/
Additionally my RSS feed changed to - http://blogs.msdn.com/b/jaredpar/rss.aspx
Several of my recent posts were deleted during the upgrade. I will be reposting them today but they will unfortunately re-appear in your RSS reader

All of the old URL’s should continue to work for the foreseeable future. But I imagine at some point the redirects will be removed so please take a minute and update your readers.

Nothing Is Private In The Debugger Part 2

2010-05-19T00:00:00+00:00

In a previous post I discussed how accessibility is ignored when evaluating expressions in the debugger and the unexpected scenarios that it creates. One case I neglected to mention in that article is how this behavior works with the VB late binding engine.

The expression evaluator only relaxes accessibility rules when binding an expression. This is possible because the expression evaluator effectively hosts the compiler and can override items like accessibility checks.

In the case of late binding the compiler only participates in building an expression that will call into the VB runtime late binding engine. The accessibility determination for the target of late bound call occurs in the VB runtime and is not (currently) overidable by the expression evaluator.

Late bound access combined with static access can lead to additional confusing behavior. For example.

Class C1
    Private Field1 As Integer
    Public Property Property1 As Integer
End Class

Module Module1

    Sub Main()
        Dim v1 As New C1
        Dim v2 As Object = v1
        Stop
    End Sub

End Module

When the following code is run, all members are accessible from the v1 local. It is statically typed to C1 and hence the expression evaluator can ignore accessibility and access the values.

image

The local v2 references the same object instance so it’s reasonable to assume it can access the same values. However because it’s statically typed as object, calls like v1.Field1 actually turn into late bound calls and hence are subject to the rules of the late binder.

image

Evaluation of Field1 fails here because the late binder does not allow access to private fields. Property1 evaluates just fine though because it’s public.

We are considering changing this behavior in a future release. As usual no promises.

The Debugger Is Different

2010-05-17T00:00:00+00:00

The goal of the debugger is to provide rich inspection capabilities for a process. The main method of inspection is through the evaluation of textual expressions which is handled by a language specific component known as the expression evaluator. This component is the data provider for a good portion of the debugger windows (watch, locals, autos, etc ‘)

The expression evaluators go to great lengths to ensure that expression in the debugger evaluate exactly as it would if the expression was typed at the current place in the file where the debugger was stopped. To do otherwise only leads to user confusion. Often making it harder to track down the issue you’re currently debugging and resulting in a loss in faith in the quality of the debugger and language.

Occasionally, or not so occasionally, the goals of providing high fidelity in evaluation and rich inspection conflict. When this occurs we have to weight the tradeoffs of confusing users by changing the semantics of the language vs. the resulting increased capabilities in the debugger.

One example is accessibility. Languages have strong notions of accessibility which is enforced by the compiler and CLR. In order to provide rich inspection though the debugger must be able to access all available data including items which are not accessible by the language. To do otherwise might arbitrarily hide that one piece of data a developer needs to solve the problem at hand.’? Hence accessibility is one case where the expression evaluators bend language rules and allow expressions to access data without accessibility checks.

One the surface this doesn’t seem like a big change. It’s an additive change allowing users to see more fields, properties and methods. In the majority case it’s as simple as that and leads to little confusion.

However even seemingly simple changes like removing accessibility checks can lead to very surprising behavior for our users. Recently our QA team filed a bug that illustrates this point.

    Module Module1
        Class Base
            Private Field1 As Integer = 55
        End Class
    
        Class Derived
            Inherits Base
            Sub Method1()
                Field1 = 72
                Stop
            End Sub
        End Class
    
        Sub Main()
            Dim x As New Derived
            x.Method1()
        End Sub

    End Module

    Module Module2
        Public Field1 As Integer
    End Module

QA noted that if they ran this code and typed Field1 in the watch window that the value showed 55 instead of 72. This was noted as a bug because Field1 was set to 72 on the line before.

Interestingly enough though is that this behavior, while very confusing, is actually ‘By Design’.

The reason why is that when evaluating expressions in the debugger nothing is private. Knowing this reconsider what it means to evaluate Field1 inside of Method1 with both instances being Public. The correct binding is Base::Field1 since the VB language prefers instance fields over Module fields if they are both accessible. The expression evaluator correctly evaluates this as 55.

However when the code was compiled accessibility checks were in place. This mean that Base::Field1 was not considered since it was private and inaccessible. The compiler instead correctly bound to Module2::Field1 and this is the field which is used when the code is running. Developers can verify this by evaluating Module2.Field1 in any of the debugger windows.

Properly Testing Equality

2010-04-30T00:00:00+00:00

Getting equality correct on a .Net type is a fairly involved process involving adherence to a large set of rules in order to be considered correct. Including

Object.Equals overrides on reference types must return false for null values
Object.Equals overrides must return false for incompatible types
Excluding null cases x.Equals(y) must be the same as y.Equals(x)
Excluding null cases (x.Equals(y) && y.Equals(z)) is true only if x.Equals(z)
If operator == or overloaded
- Both == and != should be overloaded or none
- Operator == must handle the left side being null
- Operator == should mimic Object.Equals in all cases where the left side is not null
- Operator != must handle the left side being null
- Operator != should mimic !Object.Equals in all cases where the left side is not null
If two values are equal according to Object.Equals they must have matching returns for GetHashCode

I’m sure I missed one or two subtle ones but these are the major players. It gets even more fun when you add in IEquatable to the mix.

Luckily correctly implementing equality is fairly straight forward and most template code available on the web respects the above rules. However it’s easy to miss a corner case and add hard to track down bugs.

I’m not satisfied by simply following a standard template and hoping I got it right. I only sleep easy if I’ve tested these cases. Yet testing all of these cases is very tedious and involves quite a bit of code that screams for an abstraction. As a new type author I simply want to provide a collection of units which associate a value and corresponding equal or not equal values and let the abstraction verify I properly implemented equality semantics.

The first step is defining a type to encapsulate a value and set of equal or not equal values.

public class EqualityUnit<T> {
    private static ReadOnlyCollection<T> EmptyCollection = new ReadOnlyCollection<T>(new T[] { });

    public readonly T Value;
    public readonly ReadOnlyCollection<T> EqualValues;
    public readonly ReadOnlyCollection<T> NotEqualValues;
    public IEnumerable<T> AllValues {
        get { return Enumerable.Repeat(Value, 1).Concat(EqualValues).Concat(NotEqualValues); }
    }
    public EqualityUnit(T value) {
        Value = value;
        EqualValues = EmptyCollection;
        NotEqualValues = EmptyCollection;
    }
    public EqualityUnit(
        T value,
        ReadOnlyCollection<T> equalValues,
        ReadOnlyCollection<T> notEqualValues) {
        Value = value;
        EqualValues = equalValues;
        NotEqualValues = notEqualValues;
    }
    public EqualityUnit<T> WithEqualValues(params T[] equalValues) {
        return new EqualityUnit<T>(
            Value,
            EqualValues.Concat(equalValues).ToList().AsReadOnly(),
            NotEqualValues);
    }
    public EqualityUnit<T> WithNotEqualValues(params T[] notEqualValues) {
        return new EqualityUnit<T>(
            Value,
            EqualValues,
            NotEqualValues.Concat(notEqualValues).ToList().AsReadOnly());
    }
}

public static class EqualityUnit {
    public static EqualityUnit<T> Create<T>(T value) {
        return new EqualityUnit<T>(value);
    }
}

I chose a fluent interface design here because it makes the usage code very readable. For example

var unit = EqualityUnit
    .Create(new MyType(42))
    .WithEqualValues(new MyType(42))
    .WithNotEqualValues(new MyType(13));

Now that we have the data defined we need to follow through with the actual test code. Most of it is very straight forward enforcement of the above said rules. The only trick part is how to test operator == and !=.’? The testing class is necessarily generic but neither == or != can be used against open generic types. Instead we must use them against the non-generic types.

This can be solved by having the calling code provide 2 lambda expressions of type Func<T,T,bool> which call the == and != operator.

EqualityUtil.RunAll(
    (x, y) => x == y,
    (x, y) => x != y,

This is boiler plate code that has to be repeated for every caller but it’s small enough to not be that much of a burden.’? Now finally the code.

public sealed class EqualityUtil<T> {
    private readonly ReadOnlyCollection<EqualityUnit<T>> _equalityUnits;
    private readonly Func<T, T, bool> _compareWithEqualityOperator;
    public readonly Func<T, T, bool> _compareWithInequalityOperator;

    public EqualityUtil(
        IEnumerable<EqualityUnit<T>> equalityUnits,
        Func<T, T, bool> compEquality,
        Func<T, T, bool> compInequality) {
        _equalityUnits = equalityUnits.ToList().AsReadOnly();
        _compareWithEqualityOperator = compEquality;
        _compareWithInequalityOperator = compInequality;
    }

    public void RunAll(
        bool skipOperators = false,
        bool skipEquatable = false) {
        if (!skipOperators) {
            EqualityOperator();
            EqualityOperatorCheckNull();
            InEqualityOperator();
            InEqualityOperatorCheckNull();
        }

        if (!skipEquatable) {
            ImplementsIEquatable();
            EquatableEquals();
            EquatableEqualsCheckNull();
        }

        ObjectEquals();
        ObjectEqualsCheckNull();
        ObjectEqualsDifferentType();
        GetHashCodeSemantics();
    }

    private void EqualityOperator() {
        foreach (var unit in _equalityUnits) {
            foreach (var value in unit.EqualValues) {
                Assert.IsTrue(_compareWithEqualityOperator(unit.Value, value));
                Assert.IsTrue(_compareWithEqualityOperator(value, unit.Value));
            }

            foreach (var value in unit.NotEqualValues) {
                Assert.IsFalse(_compareWithEqualityOperator(unit.Value, value));
                Assert.IsFalse(_compareWithEqualityOperator(value, unit.Value));
            }
        }
    }

    private void EqualityOperatorCheckNull() {
        if (typeof(T).IsValueType) {
            return;
        }

        foreach (var value in _equalityUnits.SelectMany(x => x.AllValues)) {
            if (!Object.ReferenceEquals(value, null)) {
                Assert.IsFalse(_compareWithEqualityOperator(default(T), value));
                Assert.IsFalse(_compareWithEqualityOperator(value, default(T)));
            }
        }
    }

    private void InEqualityOperator() {
        foreach (var unit in _equalityUnits) {
            foreach (var value in unit.EqualValues) {
                Assert.IsFalse(_compareWithInequalityOperator(unit.Value, value));
                Assert.IsFalse(_compareWithInequalityOperator(value, unit.Value));
            }

            foreach (var value in unit.NotEqualValues) {
                Assert.IsTrue(_compareWithInequalityOperator(unit.Value, value));
                Assert.IsTrue(_compareWithInequalityOperator(value, unit.Value));
            }
        }
    }

    private void InEqualityOperatorCheckNull() {
        if (typeof(T).IsValueType) {
            return;
        }
        foreach (var value in _equalityUnits.SelectMany(x => x.AllValues)) {
            if (!Object.ReferenceEquals(value, null)) {
                Assert.IsTrue(_compareWithInequalityOperator(default(T), value));
                Assert.IsTrue(_compareWithInequalityOperator(value, default(T)));
            }
        }
    }

    private void ImplementsIEquatable() {
        var type = typeof(T);
        var targetType = typeof(IEquatable<T>);
        Assert.IsTrue(type.GetInterfaces().Contains(targetType));
    }

    private void ObjectEquals() {
        foreach (var unit in _equalityUnits) {
            var unitValue = unit.Value;
            foreach (var value in unit.EqualValues) {
                Assert.IsTrue(unitValue.Equals(value));
                Assert.IsTrue(value.Equals(unitValue));
            }
            foreach (var value in unit.NotEqualValues) {
                Assert.IsFalse(unitValue.Equals(value));
                Assert.IsFalse(value.Equals(unitValue));
            }
        }
    }

    /// <summary>
    /// Comparison with Null should be false for reference types
    /// </summary>
    private void ObjectEqualsCheckNull() {
        if (typeof(T).IsValueType) {
            return;
        }

        var allValues = _equalityUnits.SelectMany(x => x.AllValues);
        foreach (var value in allValues) {
            Assert.IsFalse(value.Equals(null));
        }
    }

    private sealed class NotAccessible { } 

    /// <summary>
    /// Passing a value of a different type should just return false
    /// </summary>
    private void ObjectEqualsDifferentType() {
        var allValues = _equalityUnits.SelectMany(x => x.AllValues);
        foreach (var value in allValues) {
            Assert.IsFalse(value.Equals(new NotAccessible()));
        }
    }

    private void GetHashCodeSemantics() {
        foreach (var unit in _equalityUnits) {
            foreach (var value in unit.EqualValues) {
                Assert.AreEqual(value.GetHashCode(), unit.Value.GetHashCode());
            }
        }
    }

    private void EquatableEquals() {
        foreach (var unit in _equalityUnits) {
            var equatableUnit = (IEquatable<T>)unit.Value;
            foreach (var value in unit.EqualValues) {
                Assert.IsTrue(equatableUnit.Equals(value));
                var equatableValue = (IEquatable<T>)value;
                Assert.IsTrue(equatableValue.Equals(unit.Value));
            }

            foreach (var value in unit.NotEqualValues) {
                Assert.IsFalse(equatableUnit.Equals(value));
                var equatableValue = (IEquatable<T>)value;
                Assert.IsFalse(equatableValue.Equals(unit.Value));
            }
        }
    }

    /// <summary>
    /// If T is a reference type, null should return false in all cases
    /// </summary>
    private void EquatableEqualsCheckNull() {
        if (typeof(T).IsValueType) {
            return;
        }

        foreach (var cur in _equalityUnits.SelectMany(x => x.AllValues)) {
            var value = (IEquatable<T>)cur;
            Assert.IsFalse(value.Equals(null));
        }
    }
}

public static class EqualityUtil {
    public static void RunAll<T>(
        Func<T, T, bool> compEqualsOperator,
        Func<T, T, bool> compNotEqualsOperator,
        bool skipOperators,
        bool skipEquatable,
        params EqualityUnit<T>[] values) {
        var util = new EqualityUtil<T>(values, compEqualsOperator, compNotEqualsOperator);
        util.RunAll(skipOperators: skipOperators, skipEquatable: skipEquatable);
    }

    public static void RunAll<T>(
        Func<T, T, bool> compEqualsOperator,
        Func<T, T, bool> compNotEqualsOperator,
        params EqualityUnit<T>[] values) {
        RunAll(compEqualsOperator, compNotEqualsOperator, skipEquatable: false, skipOperators: false, values: values);
    }
}

And what would any code, including test framework code be without a few test cases?

[TestFixture]
public class EqualityUtilTesting {

    [Test]
    public void EqualityWithIntegers() {
        EqualityUtil.RunAll(
            (x, y) => x == y,
            (x, y) => x != y,
            EqualityUnit.Create(1).WithEqualValues(1).WithNotEqualValues(2),
            EqualityUnit.Create(42).WithNotEqualValues(13));
    }

    [Test]
    public void EqualityWithStrings() {
        EqualityUtil.RunAll(
            (x, y) => x == y,
            (x, y) => x != y,
            EqualityUnit.Create("foo").WithEqualValues("foo").WithNotEqualValues("no"),
            EqualityUnit.Create("FOO").WithNotEqualValues("foo"));
    }
}

Dictionary Tkey Tvalue Trygetvalue And Anonymous Types

2010-03-23T00:00:00+00:00

One of the methods I find to be the most useful in .Net is the method Dictionary<TKey,TValue>.TryGetValue. This method is a nice compromise between performance, explicit return vs. exception, and a being verbal about the chance of failure. It returns false on failure and uses an out parameter to return the actual requested value. This leads to the following elegant pattern

Student value;
if (map.TryGetValue("SomeKey", out value)) {
    // Value is present
}
else {
    // Value is not present
}

This works great right up until have a Dictionary of anonymous types. The TryGetValue pattern functions on out parameters which do not work well with type inference in C# and hence anonymous types. Type inference requires that the value be declared with a corresponding initialization expression. But any out call forces the declaration of the type and the initialization expression to be different statements breaking any chance of type inference.

For example take the following code which builds up a Dictionary object where the value is typed to be an anonymous type ¹

var query = from it in GetStudents()
            where it.LastName.StartsWith(lastNamePrefix)
            select new { FirsName = it.FirstName, LastName = it.LastName };
// ...
var map = query.ToDictionary(x => x.FirsName);

// How to use TryGetValue?  
WhatDoIPutHere' value;
if ( map.TryGetValue("Joe", out value) {
    // ...
}

To fix this problem we need to write a wrapper around TryGetValue which allows us to combine both the presence or absence of the entry in the Dictionary and the resulting looked up value if present.’? Within our wrapper method we can use type inference tricks to avoid naming the anonymous type directly. To combine the values could construct a new type say TryGetValueResult<TValue>

struct TryGetValueResult<TValue> {
    public readonly bool Success;
    public readonly TValue Value;
    public TryGetValueResult(bool success, TValue value) {
        Success = success;
        Value = value;
    }
}

But I find this to be a bit heavy handed for a simple return. Instead I prefer to combine the data with the new Tuple<T1,T2> type introduced in 4.0.’? This type is designed to be a light weight method for combining two related values into a single instance. Perfect for this type of method.

public static Tuple<bool, TValue> TryGetValue<TKey, TValue>(
    this Dictionary<TKey, TValue> map, 
    TKey key) {

    TValue value;
    var ret = map.TryGetValue(key, out value);
    return Tuple.Create(ret, value);
}

Now that we’ve built our wrapper method we can go back to the original code sample and use it to access the anonymous type values

var tuple = map.TryGetValue("Joe");
if (tuple.Item1) {
    Console.WriteLine(tuple.Item2);
}

This pattern is not limited strictly to TryGetValue. It’s fairly applicable anytime you need to combine a return value and one or more out parameters into a single value for reasons of type inference.

Believe it or not, having a Dictionary where the value type is an anonymous type is not a wholly uncommon act. I’ve run into a bit of customer code which follows this general pattern ↩

Vsvim Update Released Version 0 7 2

2010-03-15T00:00:00+00:00

I just released an update to VsVim for Visual Studio 2010 RC. This should be available shortly from the extension manager in Visual Studio or it can be downloaded directly at the following link

Link: http://visualstudiogallery.msdn.microsoft.com/en-us/59ca71b3-a4a3-46ca-8fe1-0e90e3f79329

New Features

Key mapping: map, remap, noremap and there many variations are now supported
Reading vimrc?? files on startup
A FAQ: http://wiki.github.com/jaredpar/VsVim/faq

Future Plans

Originally I planned on moving straight from 0.7.0 to 0.8.0. However shortly after Visual Studio 2010 RC was released I saw a significant update in number of users for VsVim. With the new users came many new issues and feature requests. Several of the requests made, such as key remapping, were blocking issues for users. As such I decided to prioritize them over getting flashy new features into the project.

Going forward I intend to leave the 0.7.X branch as the way of unblocking users and to have 0.8.0 be the release for new value add features which aren’t necessarily blocking users (such as the . operator and a proper margin instead of using the status bar). I hope with this release I can unblock the majority of users and focus on getting 0.8.0 out the door. But I will release a 0.7.3 if there is further need for unblocking users.

Thanks to everyone who took the time to notify me about issues either by filing bugs on GitHub or sending me an email!

Other Notes

VsVim is now fully available on GitHub!

http://github.com/jaredpar/VsVim

In addition to source code control I am using GitHub for both Bug Tracking and a FAQ. Bug tracking is open to the public so please file any issues or feature requests there.

The usual caveats and expectations

This extension is being released by me, not by Microsoft. As such the support level for this extension is equivalent to the amount of free time I have to put into it.

Source for this release is available on the GitHub project site. It and the associated binaries are released under the MS-PL.

Flattening Class Hierarchies When Debugging C

2010-02-19T00:00:00+00:00

One piece of feedback I heard in the MVP sessions this week is that debugging deep class hierarchies in C# is painful. By default C# will only display the fields and properties declared on a given type. To get to base class members you must expand the base node. For large hierarchies this can take several rounds of expansions to get to the desired value.

Take for instance the following class hierarchy

class Animal {
    public string name;
    public Animal(string name) {
        this.name = name;
    }
}

class Dog : Animal {
    public string color;
    public Dog(string name, string color)
        : base(name) {
        this.color = color;
    }
}

class Mutt : Dog {
    public List<string> breeds;
    public Mutt(string name, string color, params string[] breeds)
        : base(name, color) {
        this.breeds = new List<string>(breeds);
    }
}

When you are working with an instance of Mutt in the debugger, it takes 3 rounds of clicking to see what it’s name is.

The bigger hierarchy the more clicks that are needed to get the the value that is desired. This can lead to a bit of frustration when you have significantly deep object hierarchies.

Several MVP’s asked for an option to flatten the hierarchies in a debugging session so they could get at their data quicker. The bad news is this option does not exist today and will not be present in Visual Studio 2010. The good news though is that you can still get this behavior in Visual Studio today by taking advantage of the existing debugging infrastructure.

In Visual Studio 2005 the debugging team added a set of attributes to the BCL which allowed end users to customize their debugging experience. We can use three of these attributes to create a flattened hierarchy for a given object.

DebuggerTypeProxy ‘ When expanding a value in the debugger instead of showing the children of the current value create an instance of this type and display it’s children instead
DebuggerBrowsable ‘ Allows the developer to manipulate the display of a single field or property in a Type
DebuggerDisplay ‘ Allows the developer to control what is displayed in the Name, Value and Type columns for instances of a Type

The basic strategy for flattening a hierarchy is to do the following in our type proxy.

Create a new type to hold the Tuple of name, value and type for every member. Lets call it Member
Add a DebuggerDisplay attribute to Member to make it’s display emulate how an equivalent member would normally be displayed in the debugger
Use reflection to grab all of the fields and properties defined for a value wrapping them in Member instances
Expose all of the Members for a value through a property as a strongly typed array
Attribute the property with DebuggerBrowsable.RootHidden. This has the effect of hiding the wrapping property and instead promoting all of it’s children in it’s place which effectively inlines all of the Member instances as children of the proxy

The result is a type proxy which will display all of the members of an object inline. Here is the full code sample

internal sealed class FlattenHierarchyProxy {

    [DebuggerDisplay("{Value}", Name = "{Name,nq}", Type = "{Type.ToString(),nq}")]
    internal struct Member {
        internal string Name;
        internal object Value;
        internal Type Type;
        internal Member(string name, object value, Type type) {
            Name = name;
            Value = value;
            Type = type;
        }
    }

    [DebuggerBrowsable(DebuggerBrowsableState.Never)]
    private readonly object _target;
    [DebuggerBrowsable(DebuggerBrowsableState.Never)]
    private Member[] _memberList;

    [DebuggerBrowsable(DebuggerBrowsableState.RootHidden)]
    internal Member[] Items {
        get {
            if (_memberList == null) {
                _memberList = BuildMemberList().ToArray();
            }
            return _memberList;
        }
    }

    public FlattenHierarchyProxy(object target) {
        _target = target;
    }

    private List<Member> BuildMemberList() {
        var list = new List<Member>();
        if ( _target == null ) {
            return list;
        }

        var flags = BindingFlags.Public | BindingFlags.NonPublic | BindingFlags.Instance;
        var type = _target.GetType();
        foreach (var field in type.GetFields(flags)) {
            var value = field.GetValue(_target);
            list.Add(new Member(field.Name, value, field.FieldType));
        }

        foreach (var prop in type.GetProperties(flags)) {
            object value = null;
            try {
                value = prop.GetValue(_target, null);
            }
            catch (Exception ex) {
                value = ex;
            }
            list.Add(new Member(prop.Name, value, prop.PropertyType));
        }

        return list;
    }
}

The last step is to attribute the root of our type hierarchy with this type proxy.

[DebuggerTypeProxy(typeof(FlattenHierarchyProxy))]
class Animal {

Now when when debugging instances which derive from Animal developers will see a flattened hierarchy of values.

Vsvim Update Released For Rc Version 0 7 1

2010-02-10T00:00:00+00:00

I just released a quick update to VsVim which moves it to the Visual Studio 2010 RC build. There are no functional changes in this release

Link: http://visualstudiogallery.msdn.microsoft.com/en-us/59ca71b3-a4a3-46ca-8fe1-0e90e3f79329

Vsvim Update Released Version 0 7 0

2010-02-04T00:00:00+00:00

I just released an update to VsVim for Visual Studio 2010 Beta2. This should be available shortly from the extension manager in Visual Studio or it can be downloaded directly at the following link

Link: http://visualstudiogallery.msdn.microsoft.com/en-us/59ca71b3-a4a3-46ca-8fe1-0e90e3f79329

New Features

Visual Mode Support
- Character, Line and Block Mode
- All movement commands supported
- Basic insert-delete operations added
Command Mode Commands - :substitute, :redo,
Support for mark navigation to global marks
Implemented almost all delete-insert operations in relevant modes
Normal Mode Commands: CTRL-R,
Expanded range parsing

Notable Bug Fixes

Issues with disabling conflicting key bindings on startup
Issues with not properly intercepting standard VS commands and routing them to Vim
Timing issue on startup that would cause certain components to not get created

Source Code

Source for this release is available at the following location. It is released under the MS-PL.

http://cid-dc25b20f65f628f8.skydrive.live.com/self.aspx/Public/VsVim/VsVim-0.7.0.zip

Future Plans

The next planned major update is version 0.8.0. It will consist of the following updates

Expanded set of commands supported in all modes
The . operator in normal mode
Ability for users to change settings. Settings supported is baked into VsVim I just haven’t exposed it via command mode or any UI
Better UI in general. Right now I use MessageBox.Show for any UI I need to display other than status bar updates. I would like to expand that to include a more configurable UI

I also expect to release a 0.7.1 version once the Visual Studio RC goes public. As part of releasing RC, all Beta2 extensions will be unpublished in the gallery and I’ll need to release an RC compatible version. At that time I will provide a link to the Beta2 binaries for those not upgrading to the RC.

The usual caveats and expectations

This extension is being released by me, not by Microsoft. As such the support level for this extension is equivalent to the amount of free time I have to put into it.

Having Fun with Events in F#

2010-02-03T00:00:00+00:00

Recently I ran into a situation where I needed to handle some events in F# in a special way. In this particular case I wanted to be able to disable and re- enable my handler based on changes in the program. Essentially the C# equivalent of continually adding and removing the handlers.

I started by using the F# Observable pattern. Disposing of the handler when I was through with it and recreating it on demand. This works great but after several uses I decided to write a full abstraction for it.’? For lack of a better name I call it ToggleHandler.

[<AbstractClass>]
type internal ToggleHandler() =
    abstract IsHandling : bool
    abstract Add : unit -> unit
    abstract Remove : unit -> unit
   
    static member Create<'T> (source:System.IObservable<'T>) (func: 'T -> unit) = ToggleHandler<'T>(source,func)
    static member Empty = 
        { new ToggleHandler() with 
            member x.Add() = ()
            member x.Remove() = () 
            member x.IsHandling = false }

and internal ToggleHandler<'T> 
    ( 
        _source : System.IObservable<'T>,
        _func : 'T -> unit) =  
    inherit ToggleHandler()
    let mutable _handler : System.IDisposable option = None
    override x.IsHandling = Option.isSome _handler
    override x.Add() = 
        match _handler with
        | Some(_) -> failwith "Already subcribed"
        | None -> _handler <- _source |> Observable.subscribe _func |> Option.Some
    override x.Remove() =
        match _handler with
        | Some(actual) -> 
            actual.Dispose()
            _handler <- None
        | None -> ()

The design goal was to support my standard pattern for consuming events. Typically I store all event handlers as let bindings within a type but the actual delegate handling the event is bound to a member. Member declarations are not available in let bindings so creating an event handler becomes a 2 step process: defining in the let binding and then actually creating inside of a do binding. ToggleHandler facilitates this by providing a very easy let binding story.

let mutable _clickHandler = ToggleHandler.Empty

The base class ToggleHandler is type independent so this will work for any event type. Creating the real binding inside of the initial do binding is likewise as easy (and lacking explicit types).

do
    _clickHandler <- ToggleHandler.Create _button.Click this.OnButtonClick
    _clickHandler.Add()

Now I can toggle my event handler at any point in the application by calling Add/Remove.

Full Sample:

    type Form1() as this =
        inherit Form()
    
        let _button = new Button()
        let mutable _clickHandler = ToggleHandler.Empty
    
        do
            _clickHandler <- ToggleHandler.Create _button.Click this.OnButtonClick
            _clickHandler.Add()
    
        member private x.OnButtonClick (e:System.EventArgs) = 
            // Handle Click 
            ()
    
        member private x.ToggleHandler() =  
            if _clickHandler.IsHandling then _clickHandler.Remove()
            else _clickHandler.Add()

Easier Script Deployment In Powershell 2 0

2010-02-02T00:00:00+00:00

If you can’t tell from reading entries in my blog I’m a bit of a script junkie. I loathe typing out the same command sequence more than once. As such I go to great lengths to script as much as possible in life. I also enjoy sharing my scripts with other members of my team.

Unfortunately deployment of PowerShell scripts has long been a point of pain for me. In the early 1.0 days PowerShell wasn’t installed by default on any OS so the only people with PowerShell were other scripter’s. Deploying to the masses simply wasn’t possible without installing PowerShell which is a significant adoption barrier. Even if PowerShell was installed, the default setup forbids the execution of any script file. Allowing scripts to run in 1.0 requires both Administrative privileges and a rather awkward command sequence: set-executionpolicy RemoteSigned.

These two combined to make it nearly impossible to distribute my scripts to the masses. Other devs, after explanations, understood the problems I encountered. But the masses simply did not. The wanted a quick, one or two click solution to their problem. Running a powershell script was simply too involved.

The best solution I found was to distribute both a .cmd and a .ps1 file. The .cmd file’s job was to set the special registry key to allow script execution and then run the script. This still hurt adoption because it required administrative privileges and an existing copy of powershell on the machine.

Thankfully the story is quite a bit better in 2.0?? The primary improvement is simply adoption. PowerShell is installed by default in Windows7 and certain flavors of W2K8. Windows7 is very popular amongst my coworkers and hence I can depend on at least a PowerShell deployment when I write my scripts.

The next improvement surrounds the execution of scripts. The default is still to have script execution disabled but you no longer have to be an administrator to do so. Additionally you can specify on the PowerShell command line what the execution policy can be. This mean you can have a script execute without the user having to manually enable script execution.

The command line is still a bit awkward. But it’s easily solved by deploying both a .cmd and .ps1 file. The .cmd file simply calls the .ps1 file with the awkward command line. Users are comfortable clicking on a .cmd file and it removes all knowledge of powershell from the equation.

For example:

Fix-Setup.cmd

set SOURCE=%~dp0
set TARGET=%SOURCE%\Fix-Setup.ps1
powershell -ExecutionPolicy RemoteSigned %TARGET%

These small changes greatly increased the adoption rate of scripts within my group.

The Many Cases of ByRef

2010-01-21T00:00:00+00:00

One of the overlooked or simply misunderstood features of the VB language is calling a function which has a ByRef parameter. Most languages support only a single method of passing parameters by reference ¹, that is the scenarios directly supported by the CLR. The CLR has a lot of restrictions on the type of values it supports for ByRef parameters and these restrictions get in the way of VB’s goal to be a flexible language that strives to get out of the way of the user. Hence the compiler goes to great lengths to be flexible and support multiple avenues of ByRef passing, much beyond what the CLR natively allows.

This article will explore these different mechanisms. In order to reduce the code samples, I will be using the following 2 methods to explain the different mechanisms of ByRef Passing

Sub FunctionWithInt(ByRef p1 As Integer)
    p1 = 42
End Sub

Sub FunctionWithObject(ByRef p1 As Object, ByVal p2 As Object)
    p1 = p2
End Sub

CLR ByRef

The first is to simply use the CLR concept of passing by reference as defined by section 12.4.1.5.2 and 12.1.6.1 of the CLI specification. Any variable which meets any of the following criteria, does not require a type conversion, and is passed to a ByRef parameter will be passed directly in the CLR.

Argument of the current method
Local variable
Member Field of an object
Static Field
Array Element

No special code is needed or generated for this scenario.

Copy Back ByRef

While the CLR method of passing ByRef is very flexible, it disallows a number of useful scenarios. The most prominent of which is properties. Properties do not meet the CLR requirements for ByRef because under the hood they are simply a pair of function calls. The result of a function call cannot be directly passed by reference.

Without any language intervention this can be very confusing to users. Properties are very often simple get/set wrappers around fields and have almost the exact same usage scenarios. To the point that most users don’t see a functional difference between the two. Auto-implemented properties blur this line even further. Not being able to pass them ByRef creates an unacceptable inconsistency in their usage.

VB removes this inconsistency and allows properties to be passed by reference. This is implemented under the hood by means of a temporary variable. Temporaries are just local variables and hence can be passed by reference. The property value is assigned to a temporary which is then passed by reference and then after wards is copied back into the original property.

For example, take the following code sample

Class C1
    Public Property P1 As Integer
    Public P2 As Integer
End Class

Sub CopyBackByRef()
    Dim v1 = New C1
    FunctionWithInt(v1.P1)
End Sub

This will result in essentially the following code being generated

Sub CopyBackByRef_Explained()
    Dim v1 = New C1
    Dim vbTemp = v1.P1
    FunctionWithInt(vbTemp)
    v1.P1 = vbTemp
End Sub

This type of ByRef passing is used in the following 2 scenarios

The value is a Property containing both a getter and setter.
Passing the value to the function requires a conversion.

The first can be done with even the strictest Option settings. However #2 can only be used with Option Strict Off because it requires an implicit narrowing conversion.

Don’t Copy Back ByRef

So far we’ve only looked at scenarios where the user wants to actually see the value returned from the ByRef parameter. There are many scenarios where the language can infer the user does not care about the return value of the function. For example, what if I just want to pass a constant value?

Sub DontCopyBackByRef()
    FunctionWithInt(42)
End Sub

This code is legal in VB and represents another method of passing by ref. This is very similar to the copy back method of passing by reference. The only difference is that it never copies the value back. It essentially generates the following code

Sub DontCopyBackByRef_Explained()
    Dim vbTemp = 42
    FunctionWithInt(vbTemp)
End Sub

This type of ByRef is used in any scenario where the value being passed cannot be assigned to. For example

The result of function calls
Read Only Properties
Constant Values

Maybe Copy Back ByRef

Up until now we’ve examined cases where the compiler can examine both the value being passed and the parameter it is being passed to and make a determination about what direction the data needs to move in. What about late binding?

Sub MaybeCopyBackByRef()
    Dim v1 As Object = Me
    Dim v2 = 13
    v1.FunctionWithInt(v2)
End Sub

Here v1 is typed to object and hence FunctionWithInt is being accessed via late binding. In this case the compiler doesn’t know the actual method being invoked runtime. Hence it cannot know up front if the parameters are ByRef or ByVal and cannot make an up front decision on the variable passing mechanism.

In order to make late binding invokes flow as smoothly as normal method invokes, the compiler will generate code to conditionally update the original value based on the runtime information about the parameter. The late binder communicates this information via an array of Boolean values, one for each parameter passed to the function. The compiler will initialize this array with true for any values it knows are updatable and false for values that are not. The late binder will then examine every parameter to the function and set the corresponding index in the array to false if the method parameter is ByVal. If it is ByRef the returned value from the function will be copied back into the original parameter array.

The resulting code looks a bit like this. You can ignore all of the Nothing values as they are not important for this discussion.

Sub MaybeCopyBackByRef_Explained()
    Dim v1 As Object = Me
    Dim v2 = 13
    Dim parameters = New Object() {v2}
    Dim isByRef = New Boolean() {True}
    NewLateBinding.LateCall(v1, Nothing, "FunctionWithInt", parameters, Nothing, Nothing, isByRef, True)
    If (isByRef(0)) Then
        v2 = CInt(parameters(0))
    End If
End Sub

Starting with version 4.0, C# now supports two versions of reference passing. In addition to the one available since 1.0 the ref modifier is now optional when making an interop call to a COM object: http://blogs.msdn.com/b/samng/archive/2009/06/16/com-interop-in-c-4-0.aspx ↩

Vsvim Update Released Version 0 6 0

2010-01-04T00:00:00+00:00

I just released an update to VsVim for Visual Studio 2010 Beta2. This should be available shortly from the extension manager in Visual Studio or it can be downloaded directly at the following link

Link: <http://visualstudiogallery.msdn.microsoft.com/en-us/59ca71b3-a4a3-46ca- 8fe1-0e90e3f79329>

New Features

Normal Mode Commands: Y,Delete, Arrow Keys, gP, gp, z,zt, z., zz, z-, zb
Command Mode Commands: d[elete], <,>, j[oin], y[ank], p[ut],
Insert Mode: Fixed Intellisense window dismissal issue (see below)

Notable Bug Fixes

In previous versions intellisense didn’t play well with insert mode. In particular attempting to dismiss intellisense with the Escape key would exit insert mode but leave intellisense showing. This release fixes the behavior to instead dismiss intellisense and leave you in Insert Mode
Normal Mode O now enters insert mode upon completion
Count was not being applied to certain edit commands (x) in Normal Mode
Normal Mode ‘dw’ was deleting the line break if the word was at the end of the line

Source Code

Source for this release is available at the following location. It is released under the MS-PL.

<http://cid- dc25b20f65f628f8.skydrive.live.com/self.aspx/Public/VsVim/VsVim-0.6.0.zip>

Future Plans

The next planned major update is version 0.7.0. The two main features of this release will be the addition of Visual Mode and the :substitute command. Additionally I plan to flesh out a lot of the areas in normal, command and visual mode.

The usual caveats and expectations

This extension is being released by me, not by Microsoft. As such the support level for this extension is equivalent to the amount of free time I have to put into it.

Making F# Type Inference Friendly for C#

2009-12-15T00:00:00+00:00

One of my current hobby projects, VsVim, requires me to make a lot of calls between F# and C# projects. The core Vim engine is a pure F# solution based on Visual Studio’s new editor. It additionally has a small hosting layer and a large test bed both written in C#.

When working with the exposed core Vim engine API, I’ve found a number of generated F# constructs which are not easily accessible from C#. The problem stems from the manner in which native F# types are exposed. Many of them are generic and?? lack type inference friendly helper methods that force awkward usage patterns in C#. Most painful is the FSharpOption type because it's a type I frequently expose in APIs.

FSharpOption is the exposed type for the native F# [option](http://msdn.microsoft.com/en-us/library/dd233245$VS.100$.aspx) construct representing a value which may or may not be present. It's similar to C#'s nullable type except that it applies to all types of values. The primary operations you want to do with an FSharpOption are

Create an option with a value
Create an option without a value
Determine if it has a value
Determine if it does not have a value
Access the value

In F# using an option is an inherent part of the language and the hence the resulting code is very elegant.

let OptionExample = 
    let optionWithValue = Some(42)
    let optionWithoutValue = None
    let isSome = Option.isSome optionWithValue
    let isNone = Option.isNone optionWithoutValue
    Option.get optionWithValue

Unfortunately the equivalent C# code is not nearly so nice.

static int OptionExample() {
    var optionWithValue = new FSharpOption<int>(42);
    var optionWithoutValue = FSharpOption<int>.None;
    var isSome = FSharpOption<int>.get_IsSome(optionWithValue);
    var isNone = FSharpOption<int>.get_IsNone(optionWithValue);
    return optionWithValue.Value;
}

Too many explicit types!!! Using any explicit type with F# related code just feels wrong.

In C#, and most other .Net languages, 4 out of the 5 operations you want to do on FSharpOption require an explicit type parameter. This resulting code is a bit tedious with a simple type like int but once you get to more complex generics it can get extremely verbose. In the case of anonymous types, it’s simply not possible to use the FSharpOption without a few wrappers.

Luckily most of these can be solved by using the familiar pattern of using a non-generic class with static generic methods. These allow C# users to take advantage of the languages type inference capabilities to reduce the verbosity of the code.

public static class FSharpOption {
    public static FSharpOption<T> Create<T>(T value) {
        return new FSharpOption<T>(value);
    }
    public static bool IsSome<T>(this FSharpOption<T> opt) {
        return FSharpOption<T>.get_IsSome(opt);
    }
    public static bool IsNone<T>(this FSharpOption<T> opt) {
        return FSharpOption<T>.get_IsNone(opt);
    }
}

Now we can rewrite the original sample a bit cleaner

static int OptionExample() {
    var optionWithValue = FSharpOption.Create(42);
    var optionWithoutValue = FSharpOption<int>.None;
    var isSome = optionWithValue.IsSome();
    var isNone = optionWithoutValue.IsNone();
    return optionWithValue.Value;
}

Notice we still haven’t fixed the None case. Fixing this is a beyond the scope of what I want to write here but it is possible in certain scenarios. You can take a look at how in one of my previous blog articles: Function C# Providing an Option.

This pattern is not just limited to the FSharpOption class but can be applied to many of the generic constructs F# exports to wrap their native types. In particular FSharpFunc<T,Result> and the various FSharpChoice<> types can be made a bit friendlier with a few wrappers.

Vsvim Update Released Version 0 5 4

2009-12-14T00:00:00+00:00

I just released an update to VsVim for Visual Studio 2010 Beta2. This should be available shortly from the extension manager in Visual Studio or it can be downloaded directly at the following link

Link: http://visualstudiogallery.msdn.microsoft.com/en-us/59ca71b3-a4a3-46ca-8fe1-0e90e3f79329

Changes

This is mainly a minor bug fix release. Primarily there was a bug in the command mode jump implementation that made it flaky in big buffers. Additionally I implemented the normal mode commands r and O.

Source Code

Source for this release is available at the following location. It is released under the MS-PL.

http://cid-dc25b20f65f628f8.skydrive.live.com/self.aspx/Public/VsVim/VsVim-0.5.4.zip

The usual caveats and expectations

This extension is being released by me, not by Microsoft. As such the support level for this extension is equivalent to the amount of free time I have to put into it.

The File System is Unpredictable

2009-12-10T00:00:00+00:00

One of the more frequent questions I answer on StackOverflow is a variation of the following.

I’m doing XXX with a file, how can I know if the file exists?

The variations include verify no one else has the file open, if the file is in use, the file is not writable, etc ‘. The answer to all of these questions is unfortunately the same. Simply put you can’t. The reason why is the fundamental nature of the file system prevents such predictive operations.

The file system is a resource with multiple levels of control that is shared between all users and processes in the system. The levels of control include but are not limited to file system and sharing permissions. At any point in time any entity on the computer may change a file system object or it’s controls in any number of ways. For example

The file could be deleted
A file could be created at place one previously did not exist
Permissions could change on the file in such a way that the current process does not have access
Another process could open the file in such a way that is not conducive to sharing
The user remove the USB key containing the file
The network connection to the mapped drive could get disconnected

Or in short

The file system is best viewed as a multi-threaded object over which you have no reliable synchronization capabilities

Many developers, and APIs for that matter, though treat the file system as though it’s a static resource and assume what’s true at one point in time will be true later. Essentially using the result of one operation to predict the success or failure of another. This ignores the possibility of the above actions interweaving in between calls. It leads to code which reads well but executes badly in scenarios where more than one entity is changing the file system.

These problems are best demonstrated by a quick sample. Lets keep it simple and take a stab at a question I’ve seen a few times. The challenge is to write a function which returns all of the text from a file if it exists and an empty string if it does not. To simplify this problem lets assume permissions are not an issue, paths are properly formatted, paths point to local drives and people aren’t randomly ripping out USB keys. Using the System.IO.File APIs we may construct the following solution.

static string ReadTextOrEmpty(string path) {
    if (File.Exists(path)) {
        return File.ReadAllText(path); // Bug!!!
    } else {
        return String.Empty;
    }
}

This code reads great and at a glance looks correct but is actually fundamentally flawed. The reason why is the code changes depends on the call to File.Exist to be true for a large portion of the function. It’s being used to predict the success of the call to ReadAllText. However there is nothing stopping the file from being deleted in between these two calls. In that case the call to File.ReadAllText would throw a FileNotFoundException which is exactly what the API is trying to prevent!

This code is flawed because it’s attempting to use one piece of data to make a prediction about the future state of the file system. This is simply not possible with the way the file system is designed. It’s a shared resource with no reliable synchronization mechanism. File.Exists is much better named as File.ExistedInTheRecentPast (the name gets much worse if you consider the impact of permissions).

Knowing this, how could we write ReadTextOrEmpty in a reliable fashion’ Even though you can not make predictions on the file system the failures of operations is a finite set. So instead of attempting to predict successful conditions for the method, why not just execute the operation and deal with the consequences of failure?

static string ReadTextOrEmpty(string path) {
    try {
        return File.ReadAllText(path);
    } catch (DirectoryNotFoundException) {
        return String.Empty;
    } catch (FileNotFoundException) {
        return String.Empty;
    }
}

This implementation provides the original requested behavior. In the case the file exists, for the duration of the operation, it returns the text of the file and if not returns an empty string.

In general I find the above pattern is the best way to approach the file system. Do the operations you want and deal with the consequences of failure in the form of exceptions. To do anything else involves an unreliable prediction in which you still must handle the resulting exceptions.

If this is the case then why have File.Exist at all if the results can’t be trusted’ It depends on the level of reliability you want to achieve. In production programs I flag any File.Exist I find as a bug because reliability is a critical component. However you’ll see my personal powershell configuration scripts littered with calls to File.Exsit. Simply put because I’m a bit lazy in those scripts because critical reliability is not important when I’m updating my personal .vimrc file.

Vsvim Update Released Version 0 5 3

2009-12-07T00:00:00+00:00

I just released an update to VsVim for Visual Studio 2010 Beta2. This should be available shortly from the extension manager in Visual Studio or it can be downloaded directly at the following link

Link: <http://visualstudiogallery.msdn.microsoft.com/en-us/59ca71b3-a4a3-46ca- 8fe1-0e90e3f79329>

Changes

Normal Mode Block Cursor

This removes the red cursor in favor of a Vim block style cursor for normal mode. This is implemented as a simple adornment in the editor and in the end required much less work than I anticipated (also allowed the removal of a lot of hacky code). Thanks a lot to Noah for helping me out here.

Basic range support in command mode.

Currently limited to line number ranges, % and . (current line)

Command Mode Infrastructure Changes

This infrastructure change allowed my to correctly implement the few command mode commands I previously supported and laid the ground work for quick support of new commands

New Normal Mode Commands: gJ

This release is small on features and ordinarily wouldn’t qualify for an update. But the cursor change was so pleasant for myself that I decided to release a quick update just for that.

What’s Next?

Add command mode support for all normal mode commands that have a command mode equivalent
Resolve normal mode edits around a selection. Currently selection is ignored and it produces unexpected behavior.
More normal mode commands: In particular, Delete, gt, gT, global mark navigation
Lifetime issues around buffers and Vim’s internal object model
Navigation in a buffer which contains multiple font sizes

The usual caveats and expectations

This extension is being released by me, not by Microsoft. As such the support level for this extension is equivalent to the amount of free time I have to put into it.

Dev Connection Talk Slides And Code

2009-12-02T00:00:00+00:00

Thanks to everyone who attended my sessions at Dev Connections. I’ve posted the material for both of my talks on my SkyDrive account. This includes the slides and projects.

<http://cid- dc25b20f65f628f8.skydrive.live.com/browse.aspx/Public/DevConnections2009>

Please let me know if you have any questions or problems with the materials or any additional feedback about the sessions.

Vsvim Update Released Version 0 5 2

2009-12-01T00:00:00+00:00

I just released an update to VsVim for Visual Studio 2010 Beta2. This should be available shortly from the extension manager in Visual Studio or it can be downloaded directly at the following link Link: http://visualstudiogallery.msdn.microsoft.com/en-us/59ca71b3-a4a3-46ca-8fe1-0e90e3f79329

Changes

Removal of conflicting key bindings

On startup VsVim will now look for any key bindings which conflict with implemented Vim commands. It will then provide a message box allowing you to remove the key bindings for this session of Visual Studio. Right now this is an all or nothing removal, future versions may make this a more granular process

Normal Mode no longer intercepts all keystrokes (such as Cut and Paste)

This bug in the previous release was a result of the command routing changes that occured between Beta1 and Beta2. I believe I know have them all worked out and VsVim should only be intercepting commands it intends to process

New Normal Mode Commands
- Mark setting: m[a-zA-Z]
- Mark Jump: `[a-z], ‘[a-z]
  - This is currently limited to local marks
- Page Up, Page Down: CTRL-U,CTRL-D
- Join: j

What’s Next?

The biggest area I want to focus on for the next release is Command mode and rounding, more normal mode behavior (especially mark support) and basic range support.

At a minimum I want to expand all of the normal mode commands I have implemented to have their corresponding command mode version implemented. Join, mark, etc ‘?? This shouldn’t be a big work item. Command mode is just something I’ve been neglecting up to this point in favor of infrastructure.

The other big item I want to work on is ranges. Many command mode commands operate on ranges and it’s something I’ve yet to implement at a core level. The basic mark support I added in this release starts the process but I have a ways to go.

Also, any suggestions users have are very welcome.

The usual caveats and expectations

This extension is being released by me, not by Microsoft. As such the support level for this extension is equivalent to the amount of free time I have to put into it.

Vim Emulator Editor For Beta2 Released

2009-11-16T00:00:00+00:00

This is essentially the same release as the original but updated for some changes that occurred in the APIs between Beta1 and Beta2.

Link: http://visualstudiogallery.msdn.microsoft.com/en-us/59ca71b3-a4a3-46ca-8fe1-0e90e3f79329

The biggest change came in the way in which Visual Studio routes commands. Vim, as you can imagine, needs to participate in command routing and these changes took awhile to sort out. I believe I’ve sorted out the issues but please send me a mail / comment if you find any bugs.

Caveats and Expectations

This extension is being released by me, not by Microsoft. As such the support level for this extension is equivalent to the amount of free time I have to put into it.

What’s next?

Right now the Visual Studio command system is the biggest inhibitor to implementing new features. Each command in Visual Studio is bound to a specific key stroke and often times this conflicts with key strokes my Vim emulator needs to process.

It’s easy to implement these commands in the core Vim engine. However the actual key strokes get intercepted by Visual Studio and are not processed.

My next feature will be essentially removing these commands so that they can make it to the Vim layer. It will require a bit of a UI since I’ll be essentially changing key bindings but hopefully I can get something basic out the door so that I can start focusing on real features again.

Type Safety Issue When Assigning Ccomptr T Instances

2009-11-04T00:00:00+00:00

Recently while making a bug fix to our selection tracking code I discovered an unexpected behavior with CComPtr instances. The crux of the fix included creating a new tracking mechanism exposed via COM in the type ISelectionTracking. The old interface, lets call it IOldTracking, was a completely unrelated interface in terms of inheritance hierarchies.

As part of the fix I changed the type of a field (m_spTracking) from CComPtr to CComPtr. I searched for assignments of m_spTracking and converted them to call the new API I added as part of the fix. I didn't search terribly hard because I was depending on the compiler to catch any places I missed. ISelectionTracking and IOldTracking are incompatible types so any places I missed will show up as compilation errors.

Or so I thought

I made my fix, ran our core check-in suites without error, checked in and moved onto the next bug. A couple hours later one of our other devs emailed me and informed me my check-in was breaking our larger, slower, suite bed run because m_spTracking was NULL. After some quick debugging I found myself looking at the following chunk of code which was apparently NULL’ing out m_spTracking in the suite.

CComPtr<IOldTracking> spOldTracking;
if ( SUCCEEDED(CreateOldSelectionTracking(&spOldTracking)) ) {
m_spTracking = spOldTracking;
}

Me and the other dev were quite shocked that this compiled at all. How is it possible to assign between CComPtr and CComPtr'?? My first thought was I must have accidentally used a CComQIPtr somewhere (quickly verified that was not the case). After a bit of searching we found the cause was one of the operater=?? instances available on CComPtr. Here is the definition

template <typename Q>
T* operator=(_In_ const CComPtr<Q>& lp) throw()
{
    if( !IsEqualObject(lp) )
    {
        return static_cast<T*>(AtlComQIPtrAssign((IUnknown**)&p, lp, __uuidof(T)));
    }
    return *this;
}

This templated operator allows for assignments between CComPtr instance no matter what the type is for the left and right side. The effect is that instead of doing compile type C++ type conversion rules, it will instead rely on runtime COM polymorphic assignment rules via IUnknown::QueryInterface. This moves assignment errors from compile time to runtime for unrelated interfaces.

This is further complicated because it only applies to assignment between CComPtr’s (and derived instances). If the right hand side of the assignment is a non-smart pointer, compile time C++ conversions will apply. To demonstrate ‘

CComPtr<ISelectionTracking> spTracking;
CComPtr<IOldTracking> spOld;
...
spTracking = spOld;  // Fails at runtime
spTracking = (IOldTracking*)spOld;  // Compilation Error

What surprised me though was talking to other developers about this issue. Most agreed with me that this is a bug in CComPtr, or at least very unexpected behavior. A surprising number though did not expect this behavior but still considered it acceptable. The difference comes down whether you view CComPtr as a simple smart pointer responsible for AddRef/Release semantics or as that plus an enabler of QueryInterface style conversions. I personally view CComPtr as a simple smart pointer with know real understanding of QueryInterface style conversions and CComQIPtr as a smart pointer which respects QueryInterface style conversions.' As such this behavior is completely unexpected.

The fix in this case was pretty straight forward (use the new API) but I was still worried about how to prevent this type of problem in the future. In particular how to get the failure back to a compile time error. In the end we settled on using a stripped down version of CComPtr we already had in our code base going forward called CComPtrEx. I’ve previously blogged about about the need for this type here. It’s different from CComPtr in the following ways

Does not have the templated version of operator= and instead relies on compile time C++ conversions checks for assignment
Allows for interfaces which have multiple paths to IUnknown (can cause a compile time error in CComPtr).

Also for purposes of rigor, we temporarily commented out the CComPtr operator, recompiled our code base and verified no new errors popped up.

Speaking At Dev Connections In Las Vegas Next Week

2009-11-03T00:00:00+00:00

Next week I will be speaking at Dev Connections in Las Vegas. I will be running the following sessions

VMS02: Future Directions for Visual Basic
VMS04: Microsoft Visual Basic IDE Tips and Tricks

Both of these talks will spend a bit of time talking about all of the progress and exciting new features we’ve added in Dev10. Given I primarily work on the IDE these days, expect a bit of IDE content to work it’s way into the Future Directions talk as well.

Using F Discriminated Unions In C Beta2

2009-10-27T00:00:00+00:00

While updating my VsVim editor extensions for Beta2 ¹ I got hit by a change in the way F# exposed discriminated unions in metadata. My extension consists of a core F# component with a corresponding set of unit tests written in C#. It’s mostly API level testing and as such I use a lot of F# generated types in my C# test assembly.

In Beta1 all information which could be extracted from a discriminated type union was immediately available on the value. The underlying type presentation was less than desirable but these details were hidden by type inference and the very accessible API.’? The type wasn’t perfect because given a particular instance only the subset of the properties relevant to the union value type were valid. All others threw exceptions. But the code use of these methods and properties flowed very well.

For instance take the following F# definition

type ActionKind =
    | Mouse = 1
    | Keyboard = 2

type ActionResult =
    | Complete of (ActionKind * int)
    | Error of string
    | NeedMore of (char -> ActionResult)

The use case in C# was quite simple

[TestMethod]
public void TestActionBeta1(){
    var res = GetResult();
    Assert.IsTrue(res.IsComplete());
    Assert.AreEqual(ActionKind.Mouse, res.Complete1.Item1);
    Assert.AreEqual(42, res.Complete1.Item2);
}

Notice how no type information is necessary and the code flows quite naturally. C# type inference works great here and allows me to do what I need to do without fussing around with little stuff. The type in this case is a detail I don’t need to know about. It simply adds no value.

Discriminated Unions in Beta2 changed substantially in this area. Instead of generating the set of all values on the exposed type, there is now an inner type generated for every discriminated union value and the properties relevant to that union value are stored on the inner type. The outer type now contains only properties to determine which type of value it is (certainly an upgrade from methods!) ²

For instance in the case of ActionResult there are 3 generated inner classes: Complete, Error and NeedMore. Each one contains a single property Item which contains the associated value(s). This means to get to the value portion a cast to the inner type must be inserted!

Lets take a a look at how the above test code has to change to deal with the Beta2 generation of ActionResult.

[TestMethod]
public void TestActionBeta2() {
    var res = GetResult();
    Assert.IsTrue(res.IsComplete);
    Assert.AreEqual(ActionKind.Mouse, ((ActionResult.Complete)res).Item.Item1);
    Assert.AreEqual(42, ((ActionResult.Complete)res).Item.Item2);
}

Notice the explicit casts which must be added to access the values. This makes it impossible to rely soley on C# type inference. I must now understand the underlying type structure of discriminated unions in order to use them. This extra cast adds no real value to my code.

My C# test assembly has literally hundreds of test cases which use this pattern on F# types. I didn’t know the return type of every method and found myself hitting ‘Goto Def’ on a lot of ‘var’ instances to discover the static type, going back to the original file and inserting the cast. It was a tedious and slow process.

Eventually I settled on a different solution. For every type I exposed in F# I added a set of extension methods in the form of AsXXX where XXX represented the name of the generated inner types.’? For example

public static ActionResult.Complete AsComplete(this ActionResult res) {
    return (ActionResult.Complete)res;
}

The advantage of this approach is 2 fold

Removes the need to explicitly name types in code and hence gets back the advantages of type inference
I can now use . on any of the values and let Intellisense help me find the appropriate method to use

This extension method allows me to get closer to the Beta1 style code

[TestMethod]
public void TestActionBeta2() {
    var res = GetResult();
    Assert.IsTrue(res.IsComplete);
    Assert.AreEqual(ActionKind.Mouse, res.AsComplete().Item.Item1);
    Assert.AreEqual(42, res.AsComplete().Item.Item2);
}

With these methods and a quick series of Find / Replace calls, I was back in business.

It’s coming I promise! ↩
It also contains a handy set of factory methods for generating values but it’s not relevant to this discussion. ↩

Vim Emulator Editor Extension Released

2009-09-08T00:00:00+00:00

I just released version 0.5.0 of VsVim: a vim emulation editor extension for Visual Studio 2010 Beta1 written in F#. This is a hobby project I’ve been working on for awhile now. I expect to continue updating this release as time goes on as I use it on a daily basis and I’m interested in getting back feedback from users on it.

Link: http://visualstudiogallery.msdn.microsoft.com/en-us/59ca71b3-a4a3-46ca-8fe1-0e90e3f79329

Here’s a quick break down on the state of this project

Caveats and Expectations

This extension is being released by me, not by Microsoft. As such the support level for this extension is equivalent to the amount of free time I have to put into it.

Quality Level

I would classify this as a Beta style release. I use this extension every day and I have a very large test bed to verify functionality. There are still several known bugs (detailed below) and little quirks I’m working on. But they are mostly minor issues

What’s Implemented

At this point the engine has an Insert, Normal and Command mode.

Insert Mode
- Basic insertion layer which allows for typing. No special insert mode commands are implemented
Normal Mode.
- Movement Commands: h,j,k,l,w,b,$,^,n,*,#
- Edit Commands: x,X,d,p,P,A,u,<,>,o
- Incremental Search
Command Mode
- :e
- Jump to line
- Beginning / end of line

**Deviations **

The biggest deviation I made from a traditional VIM engine is that I am using .Net regular expressions instead of VIM style regular expressions. This allowed me to focus on getting a lot of features written vs. spending time building a regular expression engine. Getting this working will be a focus of a later release.

Another issue is the cursor. As flexible as the new editor is, one part that is very tricky is changing the appearance of the cursor. So making a block style cursor for normal mode was not done for this release. Instead I simply color the cursor red for normal mode and black for insert mode. This will be fixed in a later release.

Bugs

Below is the list of known issues for the extension. I’ve noted all bugs for which I cannot get a steady repro. If you can find one I would appreciate you emailing the steps to me.

Cursor appearance in normal mode is not a block cursor but instead a red cursor
Need Repro: Using ‘o’ in normal mode can cause the line ending to switch from \r\n to \r or \n.
Both ‘#’ and ‘*’ match partial words instead of full words
Register list is limited to standard a-z alphabet

What’s next?

Thus far I’ve been working on features which don’t conflict with existing Visual Studio key bindings. The next big release will be focusing on the infrastructure needed to integrate these commands smoothly into the core Vim engine and Visual Studio itself. This will allow me to expand the number of implemented features a great deal.

Where’s the source?

Should be released soon. Right now I’m working out where I should host this project long term. Preferably a place where users can file bugs and leave feedback.

Why No Linq In Debugger Windows

2009-08-26T00:00:00+00:00

As the owner of the VB.Net portion of the overall debugging experience, I frequently hear the request from customers to add LINQ support into the Watch / Immediate and Locals window. Virtually every other type of expression is available in the debugger windows so why leave one of the most popular ones out?

Quick Diversion: the specifics of this article are written from the point of view of the VB.Net expression evaluator. However, the limitations blocking LINQ support (in both the architecture and overall design) are very similar between VB.Net and C#.

As usual, the primary issue is cost and the cost for LINQ in the debugger windows is very high. To understand why the cost is so high though, we must start by getting a better understanding how a language service interacts with the debugging services of Visual Studio and the general philosophy around compiler features in the debugger.

Languages in Visual Studio typically provide the following major components to support the debugging experience.

Expression Evaluator (EE): This is the language specific component which provides all of the data used in the watch, locals and immediate window, data tips, conditional breakpoints and several other components. It’s primary input is an expression in string form which is converted to a value (typically an ICorDebugValue instance) and outputs a COM object capable of inspecting that value to the core debugger ¹. Everything typed into the debugger windows goes through the EE.

This component lives in the MTA of Visual Studio and has almost no interaction with the UI / STA thread.

ENC Service: This is the component which is the work horse of ENC operations. It provides rude edit detection, metadata differencing and metadata + IL generation .

This component lives in the main STA of Visual Studio

The important thing to understand about the expression evaluator is that it’s purpose is primarily to provide an expression evaluation and data inspection service. How expression evaluation works is an article in itself but suffice it to say that it converts the string to a very low level AST then walks the nodes bottom up and evaluating the expressions using the ICorDebug APIs. The EE component has no UI and is simply a data provider for the core debugger services.

The design philosophy for Both VB.Net and C# is to have the highest level of fidelity between expressions evaluated in the EE and the actual running program. To do otherwise would lead to extremely confusing results for users. When spec’ing feature support in the VB.Net EE we start from the point of 100% fidelity, determine the problems with this design (if any) and then start the difficult process of making compromises.

LINQ expressions are very different than any other expression previously allowed in the EE window because of the features interaction with metadata. All LINQ expressions require the generation or manipulation of metadata to support the underlying lambda and/or closure. Adding support for this is one of the biggest hurdles to getting LINQ (and other features) into the EE.

Evaluating a LINQ expression is actually much closer to an ENC operation than a traditional EE one.

Currently the EE’s have no capacity for generating metadata, only interpreting it. Operations which mutate or generate metadata have traditionally only been allowed via the ENC service. Getting LINQ to work in the EE with true fidelity would require at least a minimal amount of ENC feature work.

Without getting into too much detail, lets enumerate the** new** major features necessary to evaluate a LINQ expression in the EE with true fidelity to the running program. To simplify things, we’ll start by assuming there is no other LINQ expression used in the current method and the method is only being executed at most once at any given time in the process.

A metadata generation service to support the backing for closures and lambda expressions
Convert expressions typed in the EE into IL ²
ENC support for metadata to push the new metadata for closures and lambdas into the currently executing assembly
ENC support for method body IL to remove lifted variables from the current method and redirect the references inside the closure

Issues #1 and #2 are for the most part internal architecture issues and can be solved via normal processes of code base refactoring and adding new features to an existing component. I don’t mean to imply these problems are cheap (in fact they are relatively expensive) But fixing these is somewhat of an understood quantity.

Issues #3 and #4 are where the problems start. As implemented EE’s do not have capability to create or modify metadata in the running process (that is the job of the ENC service). EE’s do have access to the underlying CLR ENC APIs so it is possible to implement ENC operations in the EE. It’s just simply not been done yet.

Wait! Why not reuse our ENC implementation in the EE? Unfortunately the ENC service is currently tied heavily to our in memory IDE compiler and many other IDE / STA features. It in fact lives in a completely separate DLL, separate COM apartment and works on a different symbol table than the EE.

More fundamental though is that it’s designed for a completely different purpose. ENC is designed to track edits in live code, determining the differences and applying them to the running program. The hypothetical EE feature would be tracking expressions that modify a running DLL for which code is not guaranteed (and not likely) to be available and applying the difference to the running program. There are some similarities but the differences are significant enough to make code reuse have limited value.

Some people may wonder why it’s necessary to implement #4. Couldn’t we just avoid removing the variable from the current method and make the feature cheaper? This is possible but it would cause a significant fidelity difference in the feature. Any mutations of the local variable within the LINQ expression would not be visible on the stack frame as it would if the LINQ expression was present in the original program.

It’s easy to think of this ENC in the EE as just the same type of cost as #1 and #2 (in many ways it is). However taking advantage of the CLR ENC APIs in the EE also means that we inherit it’s limitations as well. ENC as as implemented in the CLR has many limitations which fly in the face of LINQ. In particular the following ENC limitations present major problems (ENC limitations reference).

Cannot add members to a value type. This prevents evaluating LINQ when stopped in a value type method
Cannot remove locals from a function. This is an implementation of LINQ but we could work around this problem by changing the IL to simply no longer accessing the local variables.
Cannot change anything in a generic type. This prevents evaluating LINQ when stopped in a generic type.
Cannot specify an initial value for newly added fields.
Allowable ENC operations differ between the top of the stack and operations elsewhere within the stack
Modification of a non-top stack frame cannot significantly edit the current function call. So if a LINQ expression captures a variable used in that call (think ref passing) it would likely be unable to be evaluated.

So even if we implemented everything possible on the language service side the resulting feature would be limited in several impactful ways. Additionally these limitations are somewhat orthogonal to the current limitations EE’s face and would require a bit of user education. And this is only for the most basic LINQ feature under unrealistic scenarios.

Lets now consider the problems of evaluating a LINQ expression when the current method already contains a LINQ expression that lifts at least 1 variable. Evaluating a new expression at the same scope would require the modification of the current closure to maintain fidelity as opposed to generating a new one. This brings along with it a couple more problems.

A lambda expression which uses 2 variables, 1 of which is captured in an existing lambda expression. To maintain fidelity we would have to modify an existing closure signature to contain the new variable.

Often times generated closures are generic so we would run straight into ENC limitation #3. Additionally we would need the newly added field to have the same value as it has in the current method which runs us into ENC limitation #4. Now also consider that a closure instance can live much longer than the method in which it was created. For those closures no stack value is available so what value should we give fields in that instance? Ideally it should have the value of the variable at the point the method exited but realistically that’s not possible.

A lambda expression which uses 1 variable in a method where an existing lambda expression captures that same variable and another. To maintain fidelity we would have to know about this and fake capture the second variable as well.

Now consider all of the problems above and consider the scenario where there are multiple threads and multiple instances of the current method active in the program. How to determine which closure instance belongs to which thread, and just as important which stack frame on which thread, with respect to initializing values?

None of the issues are an unsolvable problem but they do represent a significant cost to the feature. And once again even if we added all of these features to the EE, ENC limitations would significantly limit the usefulness of the resulting feature.

Does this mean that LINQ, or any future metadata requiring expression, will never be added to the debugger window? Absolutely not. We’ve just hit the difficult stage of making compromises on the level of fidelity in the feature. If you back off of true fidelity in a few small ways the resulting feature, while still very expensive, is significantly cheaper and removes many of the limitations imposed by ENC. This hypothetical feature will be discussed in my next article.

For those interested the returned interface is IDebugProperty2 ↩
Right now expressions are converted to a tree form slightly above IL and they are interpreted using the ICorDebug APIs. Converting all the way down to IL requires a lot more work ↩

Code Smell Psychic Classes

2009-06-18T00:00:00+00:00

Psychic classes have the appearance of ignoring data provided to it in an attempt to provide you with an answer they predict is better for the situation.’? It’s impossible to look at a the data provided to an instance of the class and understand what queries on the object will return because it may think of a better answer for you, or a better piece of data to look at.

This comes from an example I ran into about a month ago. I work on an IDE and naturally deal with a lot of parse trees and tokens. Parsing everything all the time is expensive so naturally the results are cached in various places for performance reasons.

While debugging one such cache I noticed some strange behavior. The cache wasn’t returning the right tree for the input it was provided. So I decided to dig into the code a bit.

This cache takes several different forms of input which has no common base class or interface. What I noticed though is that when resetting the input of the service, it would not clear the existing cache or the previous form of input. Also because of the way the code loaded certain forms of input had precedence over others. So even an explicit clear did not guarantee the ‘correct’ input was used.

The result is a service that reads well in code, but will not always act as you expect it to. The service at times will seemingly ignore all input and pick a source it thinks is better. Take the following code as an example

pCache->SetSource(pSomeFile);
ParseTree* pTree = pCache->GetTree();  

This code is very straight forward but is certainly not guaranteed to do what it appears to do.

I like to think the service is predicting the results rather than calculating them. Or better yet guessing the answer. From the perspective of a code reviewer, that’s what’s happening.

Obviously I was curious about the reason for this and did a bit of research. It’s a rather old class so I had to contact people who’d been on the team awhile back and dig through the history of the code base to understand what the purpose of this behavior was. It turns out it was done to fix a few impactful scenarios where an alternate source needed precedence over the typical source. Other devs didn’t fully understand the source semantics of the service and wrote methods that caused bad interactions. Eventually it evolved to it’s current odd state.

Thanks to Dustin for coining the term ‘psychic classes’. Other ones we considered were

Jedi Mind Trick classes: Weak name
Weatherman: It’s a prediction after all??

And yes, we fixed this issue :)

Test Itemproperty Utility Function

2009-06-12T00:00:00+00:00

I was playing around in the registry the other day and found the PowerShell API lacking in a key area. There does not appear to be a good way to detect the presence of a Registry Name/Value pair. All of the operations such as New, Delete, Rename are based off of the program knowing the presence or absence of the key before hand. This lacks symmetry with the rest of the APIs which have a test style function.

PS> test-path "some\file\path\data.txt"
True

I took a few minutes and sketched out a basic Test-ItempProperty function. It utilizes the Get-ItemProperty function and suppresses the rather loud red font error message via the ‘ErrorAction parameter (standard on all CmdLets).

function Test-ItemProperty() {
    param ( [string]$path = $(throw "Need a path"),
            [string]$name = $(throw "Need a name") )

    $temp = $null
    $temp = Get-ItemProperty -path $path -name $name -errorAction SilentlyContinue
    return $temp -ne $null
}

Now I can finish up my happy scripting for the night

PS> test-path .\IsItScriptingTime
True

Can You Bing It

2009-05-28T00:00:00+00:00

Today Microsoft announced a new search engine called bing. I’ve been dogfooding this engine internally for some time now and from my experience it’s a step up from the previous engine in both functionality and appearance. It’s definitely worth playing around with for awhile.

Announcement: http://www.microsoft.com/presspass/press/2009/may09/05-28NewSearchPR.mspx

While the functionality and accuracy of the results are certainly improved, this is not the biggest selling point for me. I find Live Search’s results to be very good and noticeably improving over time. I use it at home and work and rarely find a need to switch to google.com for anything. But functionality isn’t everything.

For me the biggest selling point is the new name. Having a search engine with a 2 word name consisting of existing words prevented it from beating google in a key aspect of my life: the vernacular. Google works great as a verb and hence allows itself to be injected into the conversation. For instance ‘Can you google it real quick’?

What to do with live search though’ How can I express that I’m using live search without being too wordy’ ‘Can you live search it’ just doesn’t have the same ring to it. For awhile I shortened the name to learching (Live sEARCHING). While that works, it doesn’t quite have a good ring to it.

Bing on the other hand works great

Can you bing it?

You can view the engine at www.bing.com and the team is also twittering at @bing.

Understanding The Is Was And Will Of Programming

2009-04-27T00:00:00+00:00

When using an API you must take care to understand not only what it returns, but also for how long the data returned will be valid. This is very important to consider because programs must make either be making decisions on valid and predictable data or have appropriate fallback mechanisms for failure.

I often find bugs because people assumed certain APIs were giving back data about the present or future, but in fact the were giving back information about the past or much shorter present. Part of the problem stems from the fact that API names are usually written in the present tense regardless of what point in time the data returned is valid. For example ‘Exists’ instead of ‘Existed’ or ‘DidExist’. The language of the API lulls developers into a false sense of security and leads to programming errors.

I like to think of these values as a question of when they are valid. For instance, the data is, was or will be valid.

Will ‘ The data return is valid now and will be valid for the remainder of the program.
Is ‘ The data is valid now and will remain valid until the program or thread takes some action to alter the underlying data source.
Was ‘ The data was valid at some point in the past but by the time the program receives the result it can no longer be relied upon to be valid.

Working with APIs in the ‘was’ category are the trickiest. Because the data we are working with is volatile we must approach them in a different way in order to provide reliable programs. Most algorithms work by validating the present or future state of the system and then executing code with the expectation of success. When working with a ‘was’ API, there is no way validate the state of system because it cannot provide reliable information about its present or future state. Instead we must execute the algorithm with the expectation of failure and spend our time considering how to account for these failures appropriately. Failure must be the expected case, and not the exception.

Working with ‘will’ and ‘is’ APIs is a bit easier. Each has a level of predictability and it’s possible to use program state to make reliable decisions and hence produce reliable results.

Lets add some substance to this conversation by considering these types of APIs and the ContainsKey method on various hashtable style collections. This API is often written in the present tense regardless of how long the data is actually valid.

A ‘will’ API is the safest to work with because the data will always be valid. This API cannot be influenced by side effects. Hence the the programmer is free to think only about the algorithm at hand. These APIs are rare and are typically associated with purely immutable data structures. This is the main type of structure for which data does not ever change. For instance consider this set of calls

var name = "bob";
ImmutableMap<string, Student> map = GetSomeImmutableMap();
if (map.ContainsKey(name)) {
    SomeOtherCall();
    var student = map[name];
}

This call to the indexer will succeed no matter what. The structure cannot be changed irrespective of what actually happens in SomeOtherCall. The ContainsKey method already asserted the key was present in the hashtable. Because the structure is immutable this assertion will be valid for the remainder of the program and hence the indexer must succeed.

An ‘is’ API is the most common type of API. It is generally associated with mutable structures completely within the control of the program / thread. The data will remain valid until it is explicitly, or much worse implicitly, invalidated by the user. In a well factored program it is possible to reason about these types of APIs but hidden side effects can get you into trouble. For instance consider the following code in the context of a single thread.

var name = "bob";
Dictionary<string, Student> map = GetSomeDictionary();
if (map.ContainsKey(name)) {
    Console.WriteLine(map[name]);
    SomeOtherCall();
    Console.WriteLine(map[name]);
}

The state of a Dictionary<TKey,TValue> structure is completely within the control of the program. As such the first call to Console.WriteLine will work fine. There is nothing which can alter the state of map between the call to ContainsKey and the actual accessing of the map in the first WriteLine call so the assertion is still valid.

However without knowing the hidden side effects of the SomeOtherCall API we cannot know the next call to WriteLine will succeed. It’s possible for SomeOtherCall to access the underlying Dictionary reference and Clear it thus making the next call fail. True a programmer must investigate this before writing the above code. But consider the opposite problem. I am the maintainer of SomeOtherCall and I get a bug assigned to me which necessitates clearing out that Dictionary. I must now consider everyone who calls me, directly or indirectly and consider if any of them have an implicit dependency on the state of the underlying Dictionary object. This can get very difficult in a large program.

As previously stated, a ‘was’ API is the most dangerous type of API. They never give back data for which you can make a reliable decision about. These APIs are associated with a data source that is not completely in the control of the current thread or program. Thus it’s possible for the data source to be altered at any point in the execution of the program / thread. Threading is a prime example of this problem.

For an example, consider SynchronizedDictionary to be an implementation of Dictionary<TKey,TValue> which uses locks internally to prevent data corruption from reads / writes on multiple threads but provides no other synchronization capabilities .

var name = "bob";
SynchronizedDictionary<string, Student> map = GetSomeSynchronizedDictionary();
if (map.ContainsKey(name)) {
    var student = map[name];
}

In this example it’s possible for the call to map[name] to fail because another thread came through and cleared the collection in between the if statement and the call to map[name]. The bug here is that the program was written as if ContainsKey gave information about the present or future but in fact only gave information about the past. This actual return of the API could be made clearer by simply altering the name of the API. A much more applicable name is ‘DidContainKey’ or ‘DidAndMayStillContainKey’. This name is much more representative of the data returned and will hopefully give a developer pause before writing code like the above.

My personal pet peeve in this category is the file system and in particular File.Exists. Notice how this name is written in the present tense indicating a successful return means the file is currently in existence. Lets ignore permissions for a minute¹ and consider only the files existence. Any program on the machine is at liberty to change the file system and can do so at any point in time. Even the user can affect the file system by doing some thing as simple as yanking out a USB thumb drive or stumbling over a network cable. Any of these operations can alter the state of the file system and hence invalidate the existence of a File with 0 action from your program. Hence File.Exists cannot ever reliably determine if a file exists, it can only determine that a file ‘did’ exist at some point in the past and may exist at some point in the future. A more representative name would be File.DidExist, or File.DidAndMayStillExist.

So when designing your APIs, take care to consider the lifetime of the data when naming them.

Once you consider permissions you find that even File.DidExist in not a representative name. It probably should be called File.DidExistAndHadSomeMeasureOfAccess. Yes, that’s a terrible name :( ↩

Questions Not To Hinge A C Interview On

2009-04-21T00:00:00+00:00

People love to chat about how to conduct a C++ interview on newsgroups. Eventually these topics will shift into a discussion about what questions a candidate must know in order for them to get a hire from a particular interview. Unfortunately these questions tend to be items like the following.

What happens when you delete NULL?
Explain how exceptions affect constructor initialization.
Which constructors are called for the following: SomeType a = SomeType(b);

Sure these questions are valid parts of the language and knowing them is a plus and demonstrates a deal of mastery over the language. But not knowing them should by no means by a deal breaker. The goal of an interview is to spot good developers who will become a contributing member of the team. Good developers are motivated and passionate problem solvers. These questions are testing little more than memorization. It’s easy to memorize rules after the fact. It’s much more difficult to teach problem solving skills.

The problem with questions like the above is that a developer could be both passionate and competent in C++ and never have come across these rules. C++ is an unbelievably huge language and can operate in many different ways where these type of situations simply don’t come up. For instance, I’ve known several very good C++ developers that spent the majority of their time working with COM. COM prefers HRESULTs to Exceptions and hence they never used exceptions in C++. Once we introduced exceptions into the code base, it took about 30 minutes to explain the rules and we were done.

C++ interviews should focus on the qualities that make a good developer: problem solving and passion. Of course a language based interview should certainly focus on the foundations of C/C+. Items like pointers, stack vs. heap and the basics of memory management. But don’t hinge the deal on little items that can be quickly taught. Otherwise you’ll end up turning away good developers.

Immutable Vs Mutable Collection Performance

2009-04-06T00:00:00+00:00

One argument I commonly hear against immutable collections is they are slow. I’ve held the opposite belief for some time but shamefully had yet to look at actual numbers on the CLR. Tonight I decided to change that by benchmarking one of my immutable collections against a mutable collection in the BCL. The goal is not to decide the overall best collection but instead to get a sense of how they perform relative to each other in certain scenarios.

For the immutable version, I chose to use ImmutableCollection. This class is a slight variation of Eric Lippert’s Double Ended Queue implementation. The core algorithm is the same but the style was changed to be more inline with other immutable collections I own. For the mutable class I chose to use the ever popular [List](http://msdn.microsoft.com/en-us/library/6sh2ey19.aspx) collection.

I chose to examine the following scenarios that I commonly use with collection style classes.

Adding to the end of the collection
Adding to the front of the collection
Removing from the end of the collection
Removing from the beginning of the collection

Each scenario was run against collections of 100, 1000 and 10000 elements. For each count, the run was executed 1000 times and the total and average time was calculated. The full code for the benchmark is available at the end of this post.

Looking at the data

Now before I get into the results, please assume the usual caveats that come with any benchmark. That is, approach it with a skeptical eye. These scenarios are obviously not something I do **exactly **in everyday programming (especially removing thousands of elements from the front of List). However they are representative of general operations that I do use. Also I find it interesting to see how the collections perform relative to each other in extreme scenarios.

Most of the results were unsurprising. Remove from end is a very simple operation on a List. It comes down to a bounds check, decrementing an index and updating a couple of internal state variables.'? Removing the end of an immutable collection requires considerable updating of the internal structure. It ends up being roughly 1 order of magnitude slower.'' Adding to the end has similar implementation and numbers.

Operations on the front of the list were significantly slower on List than ImmutableCollection for suitably large collections. This is unsurprising given that removal and insertion at the front of a List requires all of the other elements in the underlying array to be shifted up or down. This is a non-trivial operation and is evident in the benchmark.

The most interesting item however, is to look at how each collection scales. In almost all scenarios, ImmutableCollection scaled very closely to the size of the input. That is, an order of magnitude more input resulted in an order of magnitude of more time. List does not have that behavior for all scenarios. Scenarios dealing with the front of the collection saw time rises faster relative to input size. In fact there is a very dramatic jump in both front scenarios between 1000 and 1000 elements. Each case resulted in roughly 2 orders of magnitude more time.

Conclusion

Winner of each category ‘

Add to End: List
Add to Front: ImmutableCollection
Remove from End: List
Remove from Front: ImmutableCollection

No single benchmark is definitive and this one won’t change that. This benchmark says nothing about the general use of the two classes. However it can provide some insight into these specific scenarios. It also serves as some level of proof that immutable collections can have acceptable performance for these scenarios.

Data

Add to End 100 Elements
       List: Total: 00:00:00.0060473 Average: 00:00:00.0000060
  Immutable: Total: 00:00:00.0267079 Average: 00:00:00.0000267
Add to End 1000 Elements
       List: Total: 00:00:00.0337505 Average: 00:00:00.0000337
  Immutable: Total: 00:00:00.2240444 Average: 00:00:00.0002240
Add to End 10000 Elements
       List: Total: 00:00:00.4266014 Average: 00:00:00.0004266
  Immutable: Total: 00:00:02.6715789 Average: 00:00:00.0026715
Add to Front 100 Elements
       List: Total: 00:00:00.0162186 Average: 00:00:00.0000162
  Immutable: Total: 00:00:00.0213764 Average: 00:00:00.0000213
Add to Front 1000 Elements
       List: Total: 00:00:00.4028523 Average: 00:00:00.0004028
  Immutable: Total: 00:00:00.2055935 Average: 00:00:00.0002055
Add to Front 10000 Elements
       List: Total: 00:00:38.5943722 Average: 00:00:00.0385943
  Immutable: Total: 00:00:02.6212590 Average: 00:00:00.0026212
Remove From End 100 Elements
       List: Total: 00:00:00.0031299 Average: 00:00:00.0000031
  Immutable: Total: 00:00:00.0213737 Average: 00:00:00.0000213
Remove From End 1000 Elements
       List: Total: 00:00:00.0187998 Average: 00:00:00.0000187
  Immutable: Total: 00:00:00.1623739 Average: 00:00:00.0001623
Remove From End 10000 Elements
       List: Total: 00:00:00.1773381 Average: 00:00:00.0001773
  Immutable: Total: 00:00:01.9615781 Average: 00:00:00.0019615
Remove From Front 100 Elements
       List: Total: 00:00:00.0142981 Average: 00:00:00.0000142
  Immutable: Total: 00:00:00.0192679 Average: 00:00:00.0000192
Remove From Front 1000 Elements
       List: Total: 00:00:00.4407993 Average: 00:00:00.0004407
  Immutable: Total: 00:00:00.1879243 Average: 00:00:00.0001879
Remove From Front 10000 Elements
       List: Total: 00:00:39.7832085 Average: 00:00:00.0397832
  Immutable: Total: 00:00:02.2451406 Average: 00:00:00.0022451

The Code

public class Program {
    public static void ImmutableCollectionAddToEnd(List<string> list) {
        var col = ImmutableCollection<string>.Empty;
        foreach (var item in list) {
            col = col.AddBack(item);
        }
    }
    public static void ListAddToEnd(List<string> list) {
        var col = new List<string>();
        foreach (var item in list) {
            col.Add(item);
        }
    }
    public static void RunAddToEnd(int count) {
        var list = Enumerable.Range(0, count).Select(x => x.ToString()).ToList();
        Console.WriteLine("Add to End {0} Elements", count);
        RunScenario("List", ListAddToEnd, () => list);
        RunScenario("Immutable", ImmutableCollectionAddToEnd, () => list);
    }
    public static void ImmutableCollectionAddToFront(List<string> list) {
        var col = ImmutableCollection<string>.Empty;
        foreach (var item in list) {
            col = col.AddFront(item);
        }
    }
    public static void ListAddToFront(List<string> list) {
        var col = new List<string>();
        foreach (var item in list) {
            col.Insert(0, item);
        }
    }
    public static void RunAddToFront(int count) {
        var list = Enumerable.Range(0, count).Select(x => x.ToString()).ToList();
        Console.WriteLine("Add to Front {0} Elements", count);
        RunScenario("List", ListAddToFront, () => list);
        RunScenario("Immutable", ImmutableCollectionAddToFront, () => list);
    }
    public static void ImmutableCollectionRemoveFromEnd(ImmutableCollection<string> col) {
        while (!col.IsEmpty) {
            col = col.RemoveBack();
        }
    }
    public static void ListRemoveFromEnd(List<string> list) {
        while (list.Count > 0) {
            list.RemoveAt(list.Count - 1);
        }
    }
    public static void RunRemoveFromEnd(int count) {
        Func<List<string>> listInputFunc = () => Enumerable.Range(0, count).Select(x => x.ToString()).ToList();
        Func<ImmutableCollection<string>> colInputFunc = () => ImmutableCollection.Create(listInputFunc());
        Console.WriteLine("Remove From End {0} Elements", count);
        RunScenario("List", ListRemoveFromEnd, listInputFunc);
        RunScenario("Immutable", ImmutableCollectionRemoveFromEnd, colInputFunc);
    }
    public static void ImmutableCollectionRemoveFromFront(ImmutableCollection<string> col) {
        while (!col.IsEmpty) {
            col = col.RemoveFront();
        }
    }
    public static void ListRemoveFromFront(List<string> col) {
        while (col.Count > 0) {
            col.RemoveAt(0);
        }
    }
    public static void RunRemoveFromFront(int count) {
        Func<List<string>> listInputFunc = () => Enumerable.Range(0, count).Select(x => x.ToString()).ToList();
        Func<ImmutableCollection<string>> colInputFunc = () => ImmutableCollection.Create(listInputFunc());
        Console.WriteLine("Remove From Front {0} Elements", count);
        RunScenario("List", ListRemoveFromFront, listInputFunc);
        RunScenario("Immutable", ImmutableCollectionRemoveFromFront, colInputFunc);
    }
    public static void RunScenario<T>(string description, Action<T> del, Func<T> getInputFunc) {
        // Run once to jit
        del(getInputFunc());
        const int times = 1000;
        var total = new TimeSpan();
        for (var i = 0; i < times; i++) {
            // get the input outside the timer so input creation is not calculated 
            var input = getInputFunc();
            var watch = new Stopwatch();
            watch.Start();
            del(input);
            watch.Stop();
            total += watch.Elapsed;
        }
        var average = TimeSpan.FromTicks(total.Ticks / times);
        Console.WriteLine("{0,11}: Total: {1} Average: {2}", description, total, average);
    }
    static void Main(string[] args) {
        var list = new int[] { 100, 1000, 10000 };
        list.ForEach(RunAddToEnd);
        list.ForEach(RunAddToFront);
        list.ForEach(RunRemoveFromEnd);
        list.ForEach(RunRemoveFromFront);
    }
}

Is It Serializable

2009-03-31T00:00:00+00:00

I’ve recently run across several APIs that have a dependency on only dealing with objects that are serializable (in the binary sense). Unfortunately determining if an object is serializable is a non-trivial task and rife with problems. These problems have a direct impact on the types of guarantees these APIs can make.

For all objects which are serializable, it’s only possible to prove that a very small subset of them actually are in code. Easier but less reliable tests are very easy to write. So APIs must make a trade off. Only accept instances of types which are provable serializable and miss out on a while class of objects. Or do a much less reliable check and open themselves up to failure further down in the algorithm.

Take System.Exception for example. It is possible to associate arbitrary data with an exception through the Data property. Associated just any object with Exception is problematic though because Exceptions should be serializable. In order for an Exception instance to store these objects and remain serializable, the objects must also be serializable. Since serializability is not provable, the authors of Exception had to make a trade off between an overly restrictive test, or a loose test. They chose the latter. As a result it’s impossible to determine before hand if a given Exception instance is actually serializable.

Why is this the case though that serialization is tough to determine’ Lets start with what it takes to make a type serializable. There are two separate components

Declaring that the type is Serializable by either having the SerializableAttribute on the class definition or by implementing ISerializable
Making the type conform to the rules of serialization.

These are completely separate actions. It’s possible to have types which do any combination of the above but not both. Take for instance the following type declarations

[Serializable]
class DeclaredOnly {
    private ConformsOnly m_conforms;
}

class ConformsOnly {
    private string m_name;
}

Both of these types are legal C# code and both represent one of the two extremes listed above. Yet neither of these types are actually serializable. ConformsOnly is not because it has not actually declared itself to be serializable. DeclaredOnly is not because one of it’s members is not serializable.

Lets look at proving serialization by ensuring types follow both of the rules. Proving the first part of serialization is pretty straight forward. Simply check to see if a type implements ISerializable or is decorated with the Serialization attribute. The latter is directly supported in the type system via Type.IsSerializable. This property is also the source of the most common mistake I see with respect to determining if an object is serializable. Take the following code snippet for an example.

public static voidExample1(object o) {  
    if(o.GetType().IsSerializable) {  
        // Do something different  
    }  
}

On the surface, this looks like reasonable code. But as we just pointed out, the property IsSerializable just determines the presence or absence of the Serializable attribute but nothing about the second part. A more descriptive attribute name would be IsSerializableAttributeDeclared. Yet many pieces of code attempt to equate this property with the ability to be serialized (A fun experiment here is to search for it’s use in Reflector)

Proving the second part involves two cases, types implementing ISerializable and types decorated with the Serializable attribute. Lets start with the attribute. Proving these is involved but a straight forward process. The type must ‘

Be decorated with the Serializable Attribute
One of the following items must be true for every field at all points in the hierarchy 1. It must be decorated with the NonSerializedAttribute 2. The type of the field must be sealed and must conform to all of these rules

Instances of types which meet these guidelines will always be serializable. Not meeting these rules though does not exclude a type from serialization. There are several sets of types decorated with Serializable which are serializable and do not meet these rules.

Take for instance types that violate rule 2.2. By having a field whose type is not sealed, it is possible to construct a runtime instance which contains a value whose type is not serializable. The following type fits into this category.

[Serializable]
class OnlyKnownPerInstance {
    private object m_field1;
}

Whether or not an instance of this type is serializable depends on the value of m_field1. So the only way to prove it is serializable is to look at the runtime information. This makes any definitive analysis on the type impossible. The actual object must be inspected.

The other case to examine are types implementing ISerializable. Serialization is a custom task for instances of these types and is done in imperative code. Proving these types are serializable involves actual algorithm inspection and is beyond the scope of this blog post. But suffice to say, proving these are serializable is an order of magnitude more difficult.

Getting back to the crux of this article. What is the best way to determine if an object is serializable or not’ Bottom line, there is no good way. The only 100% definitive way is to serialize the object and see if it succeeds or not. This is problematic because it is not future proof. It only tells you that the object **was **serializable. This is a very important distinction. It’s possible for the object to be mutated in a different state later on which will prevent it from being properly serializable.

If serialization of a parameter is very important to the semantics of an API this is the only way to ensure the semantics are not violated is to serialize the object immediately and store the binary data. Otherwise you can only make a loose guarantee that an attempt to serialize in the future will succeed.

Building A Weakreference Hashtable

2009-03-03T00:00:00+00:00

Recently I ran into a situation on a personal project where I needed a hashtable like structure for a set of WeakReference values. When poking around for an existing implementation I saw found several versions which were very thin, type safe wrapper around a Dictionary<TKey,WeakReference> (usually the class even implements IDictionary<TKey,TValue> directly). While this produces a type safe API it fails to take into account the different nature of a WeakReference. Because a WeakReference is constantly being collected without explicit user action it alters the types of operations that be performed on them. Failing to take this into account produced APIs which lead users to write incorrect code.

Finding no suitable implementation I set off to build my own. It took several iterations and I thought the result and process would be fun to share the design experience as a blog post.

Let start with the basics. At a high level the API should appear to the user as a type safe Dictionary<TKey,TValue>. Under the hood all values will be stored in an instance of WeakReference in order to enable collection. But this is an implementation detail and should not be visible to the user.’? The user should only see type safe keys and values.

A standard Hashtable works on the concept of a key/value pair. A value is associated with a particular key in the table and at any time, the value can be retrieved from the hashtable with the specified key. A key can be determined to be valid simply by ascertaining it’s presence in the underlying table. The value is irrelevant, it’s mere presence makes it valid.

A weak hashtable will work on the same concept but have a much different implementation. Keys are only valid if they point to an actual value. Since the value in the hashtable is a WeakReference the mere presence of the key does not determine it’s validity. Only the presence of the key and the value contained within the weak reference determines the validity of a key.

This seems like an obvious assumption but it has a dramatic impact on the type of API a weak hashtable can have. Lets take a simple property such as Count for an example of why this is important. Count on a hashtable is used to determine the number of valid key/value pairs in the table. On a normal hashtable, this count is simply incremented and decremented with the corresponding Add and Remove API’s. With a weak hashtable, any given run of the garbage collector can affect the count of key/value pairs by collecting a value. This means a simple counter cannot be used to keep track of the valid key/value pairs.

In order to get the actual Count every singe value must be accessed an verified that it is still alive. What’s even worse is that once a value is determined to exist, it must be stored for the duration of the Count method. Otherwise a GC could occur in the middle of the loop and collect Values that were marked as still alive.

This is what Count would need to look like ‘

public class WeakHashtable<TKey,TValue> {
    private Dictionary<TKey, WeakReference> _map;
    public int Count {
        get {
            var list = _map.Values
                .Select(x => x.Target)
                .Where(x => x != null)
                .ToList();
            return list.Count;
        }
    }
}

Count transformed from a simple O(1) return of an internal counter to a O(N) method which allocates memory. Worse yet, the return value is practically useless. As soon as the value is returned it cannot be considered to be valid. A GC could kick in and invalidate half the table. Count would in fact be giving the user information about the object in the past.

In some ways, this problem is similar to issues encountered with multi-threaded applications. In between every line of your code there is another operation, in this case the GC, which can alter the state of your structure.

The only API’s that can ask questions about a value and still have a reasonable use by the user must return the value with the call. Returning the value will, at least temporarily, provide a GC root and prevent the object from being collected. It gives the user a chance to use the value before it’s taken out from under them.

A good API comparison here are operations such as Contains and TryGetValue. Contains holds no value to the user because as soon as the call returns the GC can collect the value. TryGetValue on the other hand returns the value in question thus locking it in memory and preventing a collection.

When designing the API for a weak hashtable I tried to keep it simple and stick to these ideas. I started with the IDictionary<TKey,TValue> interface and removed the methods which hold no value for the end user due to GC limitations. In the end I was left with only the following.

void Add(TKey key, TValue value)
bool Remove(TKey key)
void Clear()
bool TryGetValue(TKey key, out TValue value)
List Values { get; }

I also added the following methods

List<Tuple<TKey,TValue» Pairs?? { get; }
Put(TKey key, TValue value)
Option TryGetValue(TKey key);

The Values property returns a List implementation instead of IEnumerable?? In order to guarantee the values remain in existence they must be rooted in some structure. The easy choice is a List. Since a List must be created anyways, why return a less accessible interface such as IEnumerable?

At first I did consider a design where Values returned IEnumerable. It is fairly simple to implement with a C# iterator.

public IEnumerable<TValue> Values {
    get {
        foreach (var weakRef in _map.Values) {
            var obj = weakRef.Target;
            if (obj != null) {
                yield return (TValue)obj;
            }
        }
    }
}

The problem though is that anything more than a simple .ForEach() over the IEnumerable may behave unexpectedly. Consecutive calls to GetEnumerator can produce different enumerations with no explicit user alteration of the table. I’ve seen several APIs which (rightly or wrongly) make this assumption. Given the user is not explicitly modifying the collection, it is not a necessarily bad assumption to make. However it would not work for a collection of this type.

I intentionally left off Keys here. Keys are only valid when they have a live value in the table. Unless the Value is returned with Key this cannot be guaranteed. The Pairs property serves this role.

This post went a bit longer than I originally intended. I also wanted to discuss how compaction of the table should work in a weak hashtable. I’ll save that for next time.

Here is the implementation of the dictionary without any compaction support.

public sealed class WeakDictionary<TKey, TValue> {
    private Dictionary<TKey, WeakReference> m_map;

    public List<Tuple<TKey, TValue>> Pairs {
        get {
            return m_map
                    .Select(p => Tuple.Create(p.Key, p.Value.Target))
                    .Where(t => t.Second != null)
                    .Select(t => Tuple.Create(t.First, (TValue)t.Second))
                    .ToList();
        }
    }

    public List<TValue> Values {
        get { return Pairs.Select(x => x.Second).ToList(); }
    }

    public WeakDictionary()
        : this(EqualityComparer<TKey>.Default) {
    }

    public WeakDictionary(IEqualityComparer<TKey> comparer) {
        m_map = new Dictionary<TKey, WeakReference>(comparer);
    }

    public void Add(TKey key, TValue value) {
        m_map.Add(key, new WeakReference(value));
    }

    public void Put(TKey key, TValue value) {
        m_map[key] = new WeakReference(value);
    }

    public bool Remove(TKey key) {
        return m_map.Remove(key);
    }

    public Option<TValue> TryGetValue(TKey key) {
        WeakReference weakRef;
        if (!m_map.TryGetValue(key, out weakRef)) {
            return Option.Empty;
        }

        var target = weakRef.Target;
        if (target == null) {
            return Option.Empty;
        }

        return (TValue)target;
    }

    public bool TryGetValue(TKey key, out TValue value) {
        var option = TryGetValue(key);
        value = option.ValueOrDefault;
        return option.HasValue;
    }
}

A More Usable Thread Safe Collection

2009-02-16T00:00:00+00:00

In my last post we discussed the problems with designing a safer API for mutable thread safe collections that employ only an internal locking system. The result was an API that was more difficult to mess up, yet pretty much unusable. Lets take a look at this problem and see if we can come up with a usable API that still helps to eliminate mistakes.

One of the main issues I have with mutable thread safe collections is the use of decision procedures such as Count and Contains. Procedures such these only return information that pertains to the collection as it existed at a previous point in time. It can provide no relevant information to the collection in it’s current state and only encourages the user to write bad code. For example.

if (col.Count > 0) {  
    // Collection can be modified before this next line executes leading to   
    // an error condition  
    var first = col[0];   
}

Therefore they have no place on a mutable thread safe collection. Yet, once you take away these procedures, you’re left with a collection that is virtually useless. It can only have a minimal API by which to access data. Here is the last example we were left with

public sealed class ThreadSafeList<T> {  
    public void Add(T value) { ... }  
    public bool TryRemove(T value) { ... }  
    public bool TryGet(int index, out T value) { ... }  
}

This is hardly a usable API. What’s worse, as wekempf point out, is that I inadvertently exposed a decision procedure in this API. It’s possible to infer state about a lower or equal index by a successful return result from TryGet(). For example, a user may say that ‘if I can access element 2, then surely element 1 must exist’. The result would still be evident in code (ignoring the return value of a TryGet method should be a red flag). But a better choice for this method would have been a TryGetFirst().

At the end of the day, users are going to want some level of determinism out of their collections. It’s possible to program against API’s like the above, but most people won’t do it. In order to be more used, the collection must be able to reliably implement procedures such as Count and Contains and allow the user to use the return to reason about the state of the collection.

One way to do this is to simply exposed the internal lock to the consumer of the collection. Consumers can take the lock and then query to their hearts content. Lets do a quick modification of the original sample to allow for this.

public sealed class ThreadSafeList<T> {  
    private List<T> m_list = new List<T>();  
    private object m_lock = new object();  
  
    public object SyncLock { get { return m_lock; } }  
  
    public void Add(T value) {  
        lock (m_lock) {  
            m_list.Add(value);  
        }  
    }  
    public void Remove(T value) {  
        lock (m_lock) {  
            m_list.Remove(value);  
        }  
    }  
    public bool Contains(T value) {  
        lock (m_lock) {  
            return m_list.Contains(value);  
        }  
    }  
    public int Count { get { lock (m_lock) { return m_list.Count; } } }  
    public T this[int index] {  
        get { lock (m_lock) { return m_list[index]; } }  
        set { lock (m_lock) { m_list[index] = value; } }  
    }  
}

Now we can go back to the original sample code and write a version which can use the decision procedures safely.

lock (col.SyncLock) {  
    if (col.Count > 0) {  
        var first = col[0];  
        ...  
    }  
}

This code will function correctly. But the API leaves a lot to be desired. In particular ‘

It provides no guidance to the user as to which procedures must be accessed with the SyncLock object locked. They can just as easily write the original bad sample code.
All procedures used within the lock reacquire the lock recursively which is definitely not advisable. We could provide properties which do not acquire the lock such as CountNoLock that work around this problem. While ok in small doses, it’s just a matter of time before you see this snippet in the middle of a huge mostly undocumented function

// Lock should be held at this point  
int count = col.CountNoLock;

This code makes my eyes bleed

The API provides 0 information to the user on exactly what the rules are for this lock. It would be left as an artifact in documentation (which you simply cannot count on users reading).
There is really nothing telling the user that they ever have to unlock the collection. Surely, any user entering into the world of threading should know this but if they do a Monitor.Enter call without a corresponding Monitor.Exit, they will receive no indication this is a bad idea.
Overall this collection requires a lot of new knowledge about the collection to use

This design though is exactly how a ‘synchronized’ collection in 1.0 version of the BCL worked. This code is essentially what you would get by passing an ArrayList instance to ArrayList.Synchronized (and most other BCL 1.0 collections). It was problematic enough that all of the new collections in 2.0 did not implement this feature. Here’s the BCL team’s explanation on this http://blogs.msdn.com/bclteam/archive/2005/03/15/396399.aspx

Overall this design poses several problems because it exposes internal implementation details directly to the consumer. An improved design should seek to hide the lock from direct access. What we really want is a way to not even provide API’s like Count and Contains unless the object is already in a locked state. This prevents them from being used at all in an incorrect scenario.

Lets run with this idea to design a more usable thread safe queue. First we’ll divide the interface for a queue into two parts.

All procedures that have 0 reliance on the internal state of the collection. Namely Enqueue, and Clear. No state is required to use these methods
All procedures that rely on the internal state of the collection to function correctly.

The ThreadSafeQueue class will contain all of the methods in category #1. It will also provide a method which returns an instance of an interface which has all of the methods in category #2.

public interface ILockedQueue<T> : IDisposable{  
    int Count { get; }  
    bool Contains(T value);  
    T Dequeue();  
}

The implementation of this interface object will acquire the internal lock of the original ThreadSafeQueue during construction and hold it for the duration if it’s lifetime. This effectively freezes the queue allowing for decision procedures to be used reliably. Implementing IDisposable and releasing the lock in the Dispose method provides a measure of lifetime management.

The rest of the code sample is below.

public sealed class ThreadSafeQueue<T> {  
  
    #region LockedQueue  
    private sealed class LockedQueue : ILockedQueue<T> {  
        private ThreadSafeQueue<T> m_outer;  
        internal LockedQueue(ThreadSafeQueue<T> outer) {  
            m_outer = outer;  
            Monitor.Enter(m_outer.m_lock);  
        }  
  
        #region ILockedQueue<T> Members  
        public int Count {   
            get { return m_outer.m_queue.Count; }  
        }  
        public bool Contains(T value) {  
            return m_outer.m_queue.Contains(value);  
        }  
        public T Dequeue() {  
            return m_outer.m_queue.Dequeue();  
        }  
        #endregion  
        #region IDisposable Members  
        public void Dispose() {  
            Dispose(true);  
            GC.SuppressFinalize(this);  
        }  
        private void Dispose(bool disposing) {  
            Debug.Assert(disposing, "ILockedQueue implementations must be explicitly disposed");   
            if (disposing) {  
                Monitor.Exit(m_outer.m_lock);  
            }  
        }  
        ~LockedQueue() {  
            Dispose(false);  
        }  
        #endregion  
    }  
    #endregion  
  
    private Queue<T> m_queue = new Queue<T>();  
    private object m_lock = new object();  
  
    public ThreadSafeQueue() { }  
    public void Enqueue(T value) {  
        lock (m_lock) {  
            m_queue.Enqueue(value);  
        }  
    }  
  
    public void Clear() {  
        lock (m_lock) {  
            m_queue.Clear();  
        }  
    }  
  
    public ILockedQueue<T> Lock() {  
        return new LockedQueue(this);  
    }  
}

This design now cleanly separates out the two modes by which the collection can be asked. It completely hides the explicit synchronization aspects from the users and replaces it with design patterns (such as IDisposable) that they are likely already familiar with. Now our original bad sample can be rewritten as follows.

static void Example1(ThreadSafeQueue<int> queue) {  
    using (var locked = queue.Lock()) {  
        if (locked.Count > 0) {  
            var first = locked.Dequeue();  
        }  
    }  
}

No explicit synchronization code is needed by the user. This design makes it much harder for the user to make incorrect assumptions or misuses of the collection. The ‘decision procedures’ are simply not available unless the collection is in a locked state.

As with most thread safe designs, there are ways in which this code can be used incorrectly

Using an instance of ILockedQueue after it's been disposed. This though is already considered taboo though and we can rely on existing user knowledge to help alleviate this problem. Additionally, static analysis tools, such as FxCop, will flag this as an error. With a bit more rigor this can also be prevented. Simply add a disposed flag and check it on entry into every method.
It’s possible for the user to maintain values, such as Count, between calls to Lock and use it to make an incorrect assumption about the state of the list.
If the user fails to dispose the ILockedQueue instance it will be forever locked. Luckily FxCop will also flag this as an error since it's an IDisposable. It's not a foolproof mechanism though.
There is nothing that explicitly says to the user ‘please only use ILockedQueue for a very short time'. IDisposable conveys this message to a point but it's certainly not perfect.
The actual ILockedQueue implementation is not thread safe. Ideally users won't pass instances of IDisposable between threads but it is something to think about.

The good news is that two of these flaws (1 and 3) are issues with existing types and tools are already designed to weed them out. FxCop will catch common cases for both of them.

Also, many of these cases are considered bad code in the absence of a thread safe collection. This allows users to rely on existing knowledge instead of forcing them to learn new design patterns for mutable thread safe collections.

Overall I feel like this design is a real win over the other versions. It provides an API which helps to limit the mistakes a user can make with a mutable thread safe collection without requiring a huge deal of new patterns in order to use.

Why Are Thread Safe Collections So Hard

2009-02-11T00:00:00+00:00

Writing a collection which is mutable, thread safe and usable is an extremely difficult process. At least that’s what you’ve likely been told all through your schooling. But then you get out on the web and see a multitude of thread safe lists, maps and queues. If it’s so hard, why are there so many examples?

The problem is there are several levels of thread safe collections. I find that when most people say thread safe collection what they really mean ‘a collection that will not be corrupted when modified and accessed from multiple threads’. Lets call this ‘data thread safe’ for brevity. This type of collection is rather easy to build. Virtually any collection that is not thread safe can be made ‘data thread safe’ by synchronizing access via a simple locking mechanism.

For Example, lets build a data thread safe List.

public sealed class ThreadSafeList<T> : IEnumerable<T> {  
    private List<T> m_list = new List<T>();  
    private object m_lock = new object();  
  
    public void Add(T value) {  
        lock (m_lock) {  
            m_list.Add(value);  
        }  
    }  
    public void Remove(T value) {  
        lock (m_lock) {  
            m_list.Remove(value);  
        }  
    }  
    public bool Contains(T value) {  
        lock (m_lock) {  
            return m_list.Contains(value);  
        }  
    }  
    public int Count { get { lock (m_lock) { return m_list.Count; } } }  
    public T this[int index] {  
        get { lock (m_lock) { return m_list[index]; } }  
        set { lock (m_lock) { m_list[index] = value; } }  
    }  
    // IEnumerable<T> left out  
}

And there you have it. The lock statement prevents concurrent access from multiple threads. So the actual m_list instance is only ever accessed by a single thread at a time which is all it’s designed to do. This means instances of ThreadSafeList can be used from any thread without fear of corrupting the underlying data.

But if building a data thread safe list is so easy, why doesn’t Microsoft add these standard collections in the framework?

Answer: ThreadSafeList is a virtually unusable class because the design leads you down the path to bad code.

The flaws in this design are not apparent until you examine how lists are commonly used. For example, take the following code which attempts to grab the first element out of the list if there is one.

static int GetFirstOrDefault(ThreadSafeList<int> list) {  
    if (list.Count > 0) {  
        return list[0];  
    }  
    return 0;  
}

This code is a classic race condition. Consider the case where there is only one element in the list. If another thread removes that element in between the if statement and the return statement, the return statement will throw an exception because it’s trying to access an invalid index in the list. Even though ThreadSafeList is data thread safe, there is nothing guaranteeing the validity of a return value of one call across the next call to the same object.

I refer to procedures like Count as decision procedures. They server only to allow you to make a decision about the underlying object. Decision procedures on a concurrent object are virtually useless. As soon as the decision is returned, you must assume the object has changed and hence you cannot use the result to take any action.

Decision procedures are one of the reasons why Immutable Collections are so attractive. They are both data thread safe and allow you to reason about there APIs. Immutable Collections don’t change ever. Hence it’s perfectly ok to have decision procedures on them because the result won’t get invalidated.

The fundamental issue with ThreadSafeList is it is designed to act like a List. Yet List is not designed for concurrent access. When building a mutable concurrent collection, in addition to considering the validity of the data, you must consider how design the API to deal with the constantly changing nature of the collection.

When designing a concurrent collection you should follow different guidelines than for a normal collection class. For example.

Don’t add an decision procedures. They lead users down the path to bad code.
Methods which query the object can always fail and the API should reflect this.

Based on that, lets look at a refined design for ThreadSafeList

public sealed class ThreadSafeList<T> {  
    private List<T> m_list = new List<T>();  
    private object m_lock = new object();  
  
    public void Add(T value) {  
        lock (m_lock) {  
            m_list.Add(value);  
        }  
    }  
    public bool TryRemove(T value) {  
        lock (m_lock) {  
            return m_list.Remove(value);  
        }  
    }  
    public bool TryGet(int index, out T value) {  
        lock ( m_lock ) {  
            if( index < m_list.Count ) {  
                value = m_list[index];  
                return true;  
            }  
            value = default(T);  
            return false;  
        }  
    }  
}

Summary of the changes

Both the Contains and Count procedures were removed because they were decision procedures
Remove was converted to TryRemove to indicate it’s potential to fail
The TryGet property was added and is reflective of the fragile nature of the method. Sure it’s possible for users to simply ignore the return value and plow on with the invalid value. However the API is not lulling the user into a false sense of security
The collection no longer implements IEnumerable. IEnumerable is only valid when a collection is not changing under the hood. There is no way to easily make this guarantee with a collection built this way and hence it was removed.
The indexers were removed as well. I’m a bit wishy washy in this particular point as there is nothing in the API which gives a user a false sense of security. But at the same time mutable concurrent collections are dangerous and should be treated with a heightened sense of respect so the indexers were removed.

This version of ThreadSafeList is more resilient, but not immune to, accidental user failure. The design tends to lead users on the path to better code.

But is it really usable? Would you use it in your application?

Simulating Closures In Powershell

2009-02-02T00:00:00+00:00

Previously I blogged about PowerShell’s lack of closure support within a script block. This presents a significant hurdle in developing a LINQ like DSL for powershell which I’ve been working on. Imagine the following syntax

$a = from it in $source where {$it -gt 5 }

This would be the rough equivalent of the following C# code

var a = from it in source where it > 5;

In C# this code works because the where clause ‘it > 5’ is converted to a lambda expression under the hood. The variable it is captured in the lambda expression via a closure. In order to get similar functionality out of powershell, the value $it must be resolvable when the ‘where’ scriptblock is executed.

Luckily Powershell is incredibly flexible. When a script block executes it will attempt to resolve any variables by looking through the various scopes. The first scope is that of the script block, and then the local scope of the code in which the script block is executed. Using new-variable, we can create variables which match the name the script block is looking for and simulate a closure.

PS) $sb = { write-host $it }
PS) & $sb

PS) new-variable "it" 42 -scope local
PS) & $sb
42

Success!!! Now all we need to do is generalize this behavior by creating a function: Run-Scriptblock. It takes two arguments

The scriptblock to execute
A list of name/value pairs. Each one represents a variable that must be available for the execution of the scriptblock

Code:

#============================================================================
# Runs a script block.  The $list parameter must be a list of string, value
# combinations.  The script block will be executed with variables of the 
# specified name and value in scope
#============================================================================
function Run-Scriptblock() {
    param ( [scriptblock] $sb = $(throw "Need a script block"), 
            [object[]]$list= $(throw "Please specify the list of names and values") )

    for ( $i = 0; $i -lt $list.Length; $i = $i+2 ) {
        $name = [string]($list[$i])
        $value = $list[$i+1]
        new-variable -name $name -value $value -scope "local"
    }

    & $sb
}

Example Usage:

PS) $sb = { write-host $it }
PS) run-scriptblock $sb "it",42
42
PS) $it

Now we have the method by which to execute a ‘where’ clause. Next time we’ll look at actually defining a LINQ DSL in powershell.

Bclextras Library

2009-01-23T00:00:00+00:00

I published a .Net utility library on Code Gallery today called BclExtras. It’s a set of classes meant to be used in addition to the standard .Net base class libraries (BCL). The main focuses of the library are functional programming, multi-threading, LINQ extensions, unit testing and API’s designed to support type inference.

This project evolved from various classes and constructs I was using in personal projects. For the last year or so I’ve kept as a separate tested library. It started out with a lot of multi-threaded code constructs but lately is leaning to a lot of functional style API’s and collections.

The library includes source and binaries for .Net 2.0 and .Net 3.5. The .Net 2.0 version of the library includes many constructs added in 3.5 that don’t rely on any 2.0SP1 CLR features. Examples are Func, Action, the Extension attribute and a subset of the LINQ Enumerable class. It allows for most LINQ expressions in a 2.0 targeted application. These types are removed in the 3.5 version to avoid conflicts with types in System.Core.

I’ve previously released this library under the name RantPack. It originally started out as personal utility library of mine and hence ended up with the somewhat obscure name. But, besides to me, RantPack doesn’t really convey a useful meaning. So I decided to give it a more meaningful name for the general population.

Powershell Linq Take Count And Take While

2009-01-16T00:00:00+00:00

The Take pair of functions are very similar to the Skip functions. The Take expression does essentially the opposite of the Skip functions. Skip is useful for getting elements further down the pipeline. Take is used for getting elements from the start of the pipeline.

#============================================================================
# Take count elements fro the pipeline 
#============================================================================
function Take-Count() {
    param ( [int]$count = $(throw "Need a count") )
    begin { 
        $total = 0;
    }
    process { 
        if ( $total -lt $count ) {
            $_
        }
        $total += 1
    }
}

#============================================================================
# Take elements from the pipeline while the predicate is true
#============================================================================
function Take-While() {
    param ( [scriptblock]$pred = $(throw "Need a predicate") )
    begin {
        $take = $true
    }
    process {
        if ( $take ) {
            $take = & $pred $_
            if ( $take ) {
                $_
            }
        }
    }
}

Example

PS) 1..10 | take-count 5
1
2
3
4
5
PS) 1..10 | take-while {$args[0] -lt 6}
1
2
3
4
5
PS)

If You Implement Iequatable T You Still Must Override Object S Equals And Gethashcode

2009-01-15T00:00:00+00:00

CLR 2.0 introduced [IEquatable](http://msdn.microsoft.com/en-us/library/ms131187.aspx) which is an interface that allows for type safe equality comparisons. Previously, the best available method for comparing equality was the virtual [Object Equals](http://msdn.microsoft.com/en-us/library/system.object.equals.aspx) method. The method is loosely typed since it takes an object as a parameter. This is easy enough to deal with on the client with a simple cast to the appropriate type.

class Student {
    public override bool Equals(object obj) {
        var other = obj as Student;
        if (other == null) {
            return false;
        }
        // rest of comparison
    }
}

IEquatable is a significant improvement over this pattern because it provides a strongly typed equals method. This protects both the caller and callee from passing incompatible object types. Additionally it avoids the overhead of boxing for value types.

These benefits are nice but if you implement IEquatable you still must override Equals(object) and GetHashCode. Not doing so is wrong and **will** cause you pain down the road. I've explored this [topic](/2008/05/09/iequatable-of-t-and-gethashcode.html) briefly in the past but wanted to expand on it a bit with some concrete examples.

Before we get into the technical details of why, lets look at this from an expectation point of view. Implementing IEquatable is a statement that 'this object knows what it means to be equal.'?? This in effect adds a contract to your class declaring that it knows how to be compared for equality. Your object should live up to these expectations in order to avoid confusing other programmers who aren't intimately familiar with your class. Confusing programmers is rarely a good idea.

**Issue #1: IEqualityComparer requires GetHashCode()**

Strongly typed collections such as Dictionary<TKey,TValue> and HashSet must be able to compare objects for equality in order to function. Starting in 2.0, the BCL provides an interface by which object equality semantics can be performed: IEqualityComparer. This class is used in many other places besides collections, but inspecting the collection classes is the easiest way to get a feel for it's use.

Lets take a look at the definition of IEqualityComparer

public interface IEqualityComparer<T> {
    bool Equals(T x, T y);
    int GetHashCode(T obj);
}

The default definition is an internal class in the BCL named GenericEqualityComparer. The default implementation of IEqualityComparer relies on IEquatable for it's implementation.

But if it uses IEquatable for it's implementation how can it possible implement GetHashCode()' Simple, it uses Object.GetHashCode(). This means an object must implement IEquatable and GetHashCode() in order to function correctly in places where IEqualityComparer is used.

But wait, I don’t actually implement IEqualityComparer anywhere so I'm safe right' Unfortunately no. Very few people actually implement IEqualityComparer. Instead they use EqualityComparere.Default to access a given IEqualityComparer for a given type T.

In fact, the standard pattern for methods which take an IEqualityComparer is to have an overload that doesn't and pass EqualityComparer.Default to the one that does.

public static class Example {
    public static IEnumerable<T> Distinct<T>(this IEnumerable<T> source) {
        return Distinct(source, EqualityComparer<T>.Default);
    }
    public static IEnumerable<T> Distinct<T>(
        this IEnumerable<T> source, 
        IEqualityComparer<T> comparer) {
        // implementation
    }
}

If your object implements IEquatable this will eventually cause it to create an instance of GenericEqualityComparer and hence a reliance on GetHashCode.

**Issue #2: Non-Strongly typed collections and Frameworks don’t use IEquatable**

IEquatable only provides equality comparisons in strongly typed scenarios. It is not convenient to access this interface in less strongly typed scenarios. Consider for instance the original 1.0 collection classes: ArrayList, Hashtable, etc '?? These are all object based collections and have no way in which to cast to IEquatable. Instead these collections must rely on the Object based methods of Equality.

Without implementing Object.Equals and Object.GetHashCode your type will not actually do any sort of comparison. This will cause lots of incorrect behavior for programmers who expect the class to understand equality.

class Person : IEquatable<Person> {
    public readonly string Name;
    public Person(string name) {
        Name = name;
    }
    public bool Equals(Person other) {
        if (other == null) {
            return false;
        }
        return StringComparer.Ordinal.Equals(Name, other.Name);
    }
}
static void EqualityCheck() {
    var p = new Person("Bob");
    var list = new ArrayList();
    list.Add(p);
    Console.WriteLine(list.Contains(p)); // Prints: True
    Console.WriteLine(list.Contains(new Person("Bob")));    // Prints: False
}

This goes against expectation. Both Person instances in this case are equal by definition of Person yet Contains fails. Implementing Object.Equals and Object.GetHashCode will remove this confusion.

The list of frameworks which still use loosely typed collections include WinForms, WPF, WebForms, etc ‘?? It’s almost inevitable that you will end up using a loosely typed collection in your project somewhere.

Isssue #3: Equality and hash codes are linked in the BCL

Rightly or wrongly, equality and hash codes are unbreakably linked in the BCL. If an object can be compared for equality it also must be able to produce a hashcode.’? This implicit contract exists many places throughout the framework. As previously displayed, implementing Object.Equals() after implementing IEquatable is straight forward and boiler plate code. Object.GetHashCode can be a bit trickier because there are many [implicit contracts](/2008/04/28/properly-implementing-equality-in-vb.html) for GetHashCode. Often mutable objects cannot provide an efficient hashing mechanism. In that case just [return 1](/2008/06/03/making-equality-easier.html). This will satisfy all of the implicit contracts around GetHashCode() and takes little time to do. Yes, it will cause a Dictionary to effectively be a linked list. But that's a heck of a lot better than simply not working at all.

Powershell Linq Skip While

2009-01-14T00:00:00+00:00

Next up in the PowerShell LINQ series is SkipWhile. This LINQ function takes an enumerable instance and a predicate. The function will skip the elements in the enumerable while the predicate is true. The argument to the predicate is the current value of the enumerable.

The LINQ version takes a predicate in the form of a Func<T,TResult>. The PowerShell equivalent of a delegate is a script block. Unlike a .Net delegate, there is no way to type the Skip-While function to accept a particular number or type of arguments. The contract with the caller will be implicit.

Other than the strict typing, the function will match the contract for the LINQ version of SkipWhile.

#============================================================================
# Skip while the condition is true
#============================================================================
function Skip-While() {
    param ( [scriptblock]$pred = $(throw "Need a predicate") )
    begin {
        $skip = $true
    }
    process {
        if ( $skip ) {
            $skip = & $pred $_
        }

        if ( -not $skip ) {
            $_
        }
    }
    end {}
}

Example Usage:

PS) 1..10 | Skip-While { $args[0] -lt 6 }
6
7
8
9
10
PS)

Linq Like Functions For Powershell Skip Count

2009-01-13T00:00:00+00:00

The PowerShell pipeline, is fairly similar to C#/VB’s LINQ. Both filter a group of elements through a series of transformations which produce a new series of elements. The devil is in the details of course but I’ll get to that in a future post.

When using PowerShell I constantly find myself wanting to use various LINQ expressions on a pipeline. Unfortunately, many LINQ expressions have no built-in equivalent in PowerShell. Most are fairly straightforward to write but a few are a bit trickier. In either case, there’s no reason for people needing to figure them out twice. So I’ll be starting a series on LINQ expressions in PowerShell.

Also, my posts are getting a bit long winded as of late. This will be a good oppuritunity to get some shorter posts up.

Today’s entry is the equivalent of Enumerable.Skip. The operation takes a count and skips ‘count’ elements in the enumeration. For PowerShell, it’s the equivalent of skipping ‘count’ elements in the pipeline.

#============================================================================
# Skip the specified number of items
#============================================================================
function Skip-Count() {
    param ( $count = $(throw "Need a count") )
    begin { 
        $i = 0
    }
    process {
        if ( $i -ge $count ) { 
            $_
        }
        $i += 1
    }
    end {}
}

Example:

PS:) 1..10 | skip-count 5
6
7
8
9
10

Dvorak Keyboard&58; Is it really faster???

2009-01-12T00:00:00+00:00

Most people discover I’m a Dvorak user because they’re in my office, attempt to drive during a conversation and find they are typing gibberish. I take the keyboard, hit CTRL+LEFT_SHIFT and they’re on their way again. Well at least until they open up a new window. It usually leads to a conversation about how I like Dvorak. And most importantly ‘ ‘is it really faster than QWERTY’?

I’ve been using a Dvorak keyboard for 4 years now. If you’re not familiar with Dvorak, it’s an alternative keyboard layout designed to be better in QWERTY in both efficiency and speed back in 1936. Other benefits are also associated with it such as ergonomics, reduced typos, etc. In my experience I find developers are much more keenly interested in the speed part.

Over 70 years later there is still a bit of debate if it achieved these goals. There are several studies of greater and lesser quality on the subject. None of which really has seemed to end the debate. If you are curious about the actual studies I encourage you to start at the wikipedia page.

Before I switched to Dvorak I read a few of the studies. In the end I’m not sure of their respective quality level but it didn’t stop me from trying out Dvorak anyways. Why? As I said there appeared to be a lot of debate, typing faster sounded nice and my former college roommate convinced me it was a great idea. The last reason being the most important.

What I didn’t see a lot of though when I was reading up on it was how the Dvorak keyboard affects the life of a developer (at least outside the speed thing). The studies were geared toward more general problems.

So this post is about how I feel the layout affects me as a developer 4 years later. Note the word feel. If you are looking for a hard core scientific study, I would navigate elsewhere. This is about my personal experiences as a developer with learning Dvorak and using it in my daily routine. So please read this post as an opinion piece and not a statement of fact.

Is it faster?

While I was reading up on Dvorak but still typing QWERTY I started timing myself with several programs. In particular I measured peak WPM and average WPM. Two weeks later I started with Dvorak and kept timing myself for about the next six months.

In terms of peak WPM Dvorak is definitely faster for me. I could get a steady peak at 120 WPM on QWERTY but can get into 140 with Dvorak.

For my normal day to day typing, I do type faster but not appreciably so. Maybe 5 WPM. I can easily hit and maintain 70-80 WPM for day to day tasks and once you’re at that speed, much more isn’t really noticed when you’re coding.

How long did it take to switch?

The first month was mostly me trying to work in Dvorak but getting so frustrated that I switched back to QWERTY to get real work done. After the first month I spent the majority of my time in Dvorak. I was noticeably slower in replying to email and occasionally I would switch back to QWERTY to fire off a longer email or code snippet. The third month was the turning point. I hit full time Dvorak and stopped switching to QWERTY except for cases where someone else was in my office.

I kept using typing tests continually check on how my overall speed was doing. Around the end of the third month I pretty much hit parity with my QWERTY abilities. After six months I did see a small increase in my average WPM and I hit 140 WPM several times.

Is it more ergonomic?

Before I started using Dvorak I was getting a bit of CTS in my right shoulder. Enough so that I started looking into exercises to help it out. The symptoms started to go away fairly soon after I started using Dvorak as my full time layout.

Now I am not actually making a claim that it’s more ergonomic. For me in my situation it helped. IMHO, I think it’s just as likely that a simple change of pace had as much to do with it as the actual layout. But if you’re desperate and you have time to kill ‘

Are any of the key repositioning impactful for development?

Yes. The repositioning of the {},[] and = keys is brutal for a developer in a C based language. Essentially anything in the top right corner of the QWERTY keyboard. They keys are in almost the same position. This made the switch even harder as muscle memory kept taking over and spitting jibberish into my code.

This will make life difficult for the first month or so.

What weird issues did I encounter?

I think the biggest issue is I’m a VIM user. For some reason I thought it a bright idea to not do any special VIM mapping and learn to use the standard VIM key mappings with Dvorak. That was ‘ a learning experience. It took quite some time before it turned back into muscle memory.

Can you still type QWERTY?

Somewhat. It takes me about 5 minutes to get going but after that I can touch type to a degree. Albeit pretty slowly. Enough though that I can use a machine for a few hours and debug a problem.

Using very cryptic command line programs such as WinDbg, PowerShell or VIM on QWERTY is very painful. It’s amazing how much of that is just muscle memory. Even if I get into QWERTY touch type mode I have to think very hard about what I’m typing in those types of programs. I flat out can’t use VIM but the rest I get by after a bit of frustration.

Do you make less typos?

That sound you heard was the rest of my coworkers and my wife laughing hysterically.

One of the benefits I read about when I switched to Dvorak is that it would reduce typos and misspellings. I’ll sum that claim up with a quick ‘Didn’t wrok for me’.

Passwords

Eventually, you will encounter a situation that forces you to type your password in QWERTY. Prior to QWERTY when choosing a new password, I would stare at my keyboard and type in gibberish for a new password. Well not exactly gibberish, it had to have a certain count of numbers, symbols, letters and a minimum length. But I would take 5 minutes and just learn the gibberish via muscle memory.

This is not so great when you need to type in a passworld via QWERTY. Muscle memory doesn’t quite translate back to the QWERTY binding. To the people sitting in the room with me I’m sure it was fairly humerous. I would type a few letters into the password box, then mouse to the desktop, type quickly, see where my fingers ended up, and then type the next few letters of the password.

The lesson is, make sure you know what your password is, and not just where to move your fingers.

Do I think that developers should switch to Dvorak?

For 95% of the developers out there I would say no. The gains are minor if any. Outside of WPM the gains are subjective. Developers really seem intent on the speed portion of the switch. This speed gain was minor for me. Once you can steady type ~70WPM the gain really doesn’t seem worth it for programming. But more importantly though, being faster won’t make you a better programmer, practice will.

Additionally, besides the learning curve of getting to Dvorak, you have to retain some level of competence at QWERTY. Whether it’s debugging problems on other peoples machines, or participating in a group presentation you will be using QWERTY more often than you think.

But there are that 5% who are

Looking for a challenge
Want to try something different
Or simply just want other people to stop using your computer

Even if you do stick it out though, the benefits are small and many are very subjective.

EDIT: Forgot the problems I had with passwords

Twitter

2009-01-09T00:00:00+00:00

I broke down a few weeks ago and joined Twitter. If you’re not familiar with Twitter, it’s a lot like sitting in a room with a bunch of people and thinking out loud. Only, it’s done in IM form. It’s interesting and I find myself musing there more and more often.

http://twitter.com/jaredpar

Most of my tweeting is programming related. But like most other people, there is the not so occasional rants and day to day ramblings.

Script Blocks And Closures Or Lack There Of

2009-01-08T00:00:00+00:00

Script blocks are a concise way of representing an expression or statement group in Powershell. It’s the C#/F#/VB lambda equivalent for PowerShell.

One difference between C#/F#/VB lambda expressions and a scriptblock is the lack of lexical closures (otherwise known as variable capturing). This feature allows for a variable defined in an outer scope to be captured by the lambda in such a way that the value is maintained with the lambda expression. The details on how the variable is captured can vary from language to language but the basics are the same.

static void ClosureExample() {
    var name = "first";
    Func<string> captureName = () => name;
    name = "second";
    Console.WriteLine(captureName());   // prints: second
}

Due to the flexible nature of powershell it is possible for a scriptblock to appear to have captured a variable when in fact it’s just a quirk of variable name resolution. An important item to remember when considering how a scriptblock will execute is knowing that a script block is evaluated at the point of execution, not the point of definition.

PS) function example1() { $b = 42; { $b } }
PS) $b = 42
PS) $sb = example1
PS) & $sb
42

The above sample works because when $sb is evaluated there is a variable $b in scope and hence the expression binds to that value. Not the original one in ‘example1’.

This is a somewhat contrived example. But the problem can easily occur when scripts 1) contain the same variable name in multiple scopes/contexts, 2) uses one of those variables within a script block. I’ve run into this problem myself several times.

Here is a more complex sample that demonstrates the timing of the name resolution.

PS) function example2() { param ($p1) $v1 = "avalue"; & $p1 }
PS) example2 {$doesnotexist}
PS) example2 {$v1}
avalue

This behavior though can also be used as a feature. Part of the implicit contract of a scriptblock can be the existance of certain named variables in the scope where the script block is executed. Probably not the best code maintainability practice, but I think we can generate a few good samples in a future post.

Script Blocks And Arguments

2009-01-07T00:00:00+00:00

Script blocks are a powershell construct for storing an expression or group of statements inside an expression. It’s the equivalent of a C#/F#/VB Lamba expression. Recently I needed to use a script block but found I had forgotten how to read passed parameters inside the script block.

Forgetting how to use a feature is typically not a blog worthy post. However this is at least the fourth time I’ve run into this situation. Hopefully blogging about it will help me remember. Or at least give me a quick search result from google.

Normally I go through a few steps when I run into an issue with PowerShell.

Use the built-in ‘‘get-help’ command
Search through my library of scripts for previous usages of a pattern
Google for a solution
Experiment within the shell

Script blocks are one of the problems that seem to hang around until step #4.

Getting to the documentation is usually a chore. It is filed under about_script_block. This is contradicts the built-in type name [scriptblock]. Usually it takes me a few tries to get the documentation up. The internal documentation is very weak for a script block. It provides little more than a basic sample.

My internal library scripts use script blocks pretty heavily but due to the incredibly terse syntax, searching for them is usually fruitless. After all, they just need a brace pair which is hardly unique within a powershell script.

This leads to step #3. I am by no means a google ninja. In fact I am often shamed by my non-technically oriented wife in getting relevant google results. Occasionally, I’ve even resorted to asking her to give me search words. Shameful, yes.

Given that with such common places words as ‘script’, ‘block’ and ‘argument’, my chances of getting a good result are pretty slim. Recently things are brightening up as more people are blogging about powershell. But it’s still hard to get a good article on the finer points of a script block.

Now we’re down to experimentation. Most scripting entry points in PowerShell have access to the built-in $args variable. This is true for both functions, scripts and filters.

PS> function test() { $args.Count }  
PS> test 42  
1  
PS> test 42 "astring"  
2

Perhaps the same is true for a script block.

PS> $a = { $args.Count }  
PS> & $a 42  
1  
PS> & $a 42 "astring"  
2  
PS>

Success!!! Maybe I’ll remember next time.

Case Sensitive Vs Case Insensitive Languages

2009-01-06T00:00:00+00:00

Just had a random thought while I was looking at some code today. It’s yet another argument in the case sensitive vs. case insensitive language battle.

Do you really want this to be able to compile?

class C1 {
  int M1;
  int m1;
}

Yes it’s an abuse and should absolutely be caught in a code review. But at the same time, experience has taught me that if an evil thing can be done with a language, someone will do it.

And yes, sadly I’ve seen code like this before.

Count Object

2008-12-19T00:00:00+00:00

With all of the great built-in commands for processing pipelines the absence of a good command to count the number of elements in a pipeline seems to stand out. The best built-in way to count the number of objects in a pipeline is to convert the value into an array and then take the length. For instance take the following script which looks for the word ‘test’ in all of the .ps1 scripts in or below the current directory

$PS> gci 're 'in *.ps1 | %{ ss test $_.FullName }

Now if i want to count that i must convert it into an array and then take the length. Simple enough right?

$PS> @(gci 're 'in *.ps1 | %{ ss test $_.FullName } ).Length

This approach works but has a couple of issues associated with it.

The first is a bit whiney I’ll admit. I often use powershell scripts in an incremental fashion. I write a search/expression, decide it includes to much data and refine it. With any decent command shell this is a pretty simple operation, hit up and add another element to the powershell pipeline and you’re in business.

The array method of counting though requires me to add data on both sides of the expression. So it’s a choice of holding down left and waiting for the cursor to get to the right position or leaving the keyboard and opting for the mouse. I don’t like either solution because it takes too long or gets my hand off of the keyboard. Did I mention this was whiney?

The second is that it forces you to allocate a contiguous block of memory to examine the?? length. While this is usually not a big concern, it can cause noticeable performance issues if you are processing a lot of data. This is especially true if the length is taken on inner expressions.

A better solution is using a filter to count the elements. Filters integrate into the powershell pipeline and process data a single element at a time. This solves both problems above. The first is that it will smoothly integrate onto the end of an existing pipeline. Also because it is a filter it processes each element individually preventing a huge array allocation.

    function Count-Object() {
        begin {
            $count = 0
        }
        process {
            $count += 1
        }
        end {
            $count
        }
    }

Now lets get back to the original problem.

$PS> gci 're 'in *.ps1 | %{ ss test $_.FullName } | count-object

Much nicer

Notimplementedexception Vs Notsupportedexception

2008-12-12T00:00:00+00:00

In responding to a recent blog post, one of the readers, Jeremy Gray, noted that I was using a NotImplementedException where I should have been using a NotSupportedException. At first I did not agree. There was a method on an interface which my underlying object could not implement therefore I felt the choice of NotImplementedException was an appropriate.

However I was also not very familiar with NotSupportedException and decided to investigate a bit more. After all, part of the fun of blogging is being wrong in a very public fashion and this was certainly a golden opportunity. The post was commenting on API design, what better way to be wrong than with a different API design issue?

After doing a bit of research I agree with Jeremy and draw the following distinction between the two exception types

NotSupportedException: Throw this exception when a type does not implement a method for which there is a corresponding property indicating whether or not the method in question is supported. For Example:
- IColletion.Add -> IsReadOnly
- Stream.Seek -> CanSeek
- Stream.Write -> CanWrite
NotImplementedException: Throw this exception when a type does not implement a method for any other reason.

For Example: ICollection.Count, ICloneable.Clone, etc … ¹

The method in question on my previous blog post was ICollection.Add(). I was dealing with an immutable collection for which Add is not possible. Since there is a property, IsReadOnly, which serves as an indicator that Add() is not allowed, [NotSupportedException](http://msdn.microsoft.com/en- us/library/system.notsupportedexception.aspx) is the better choice.

Not implementing these methods is likely a bad idea. ↩

Extension Methods And Byref Targets

2008-12-11T00:00:00+00:00

One of the seldom used, and often unknown, features of VB extension methods is that the argument of the extension method (the first parameter) can be passed by reference. This gives extension methods the power to change the reference that was used to invoke the value!

Obviously this can produce unexpected but often amusing behavior. The following sample prints ‘False’.

<Extension()> _
Public Sub EnsureNotNull(ByRef str As String)
    If str Is Nothing Then
        str = String.Empty
    End If
End Sub

Sub Example(ByVal p1 As String)
    p1.EnsureNotNull()
    Console.WriteLine("{0}", p1 Is Nothing)
End Sub

Sub Main()
    Example(Nothing)
End Sub

I think people will look at this example and either cringe, call it terrible code or get excited. Personally I’m somewhere between cringing and getting excited (in fact I’m doing both). But before people think I’ve gone off the deep end I don’t plan on checking this in any time soon (and yes I think it’s a bad idea).

Why’ The feature is fun to play with and can create some interesting samples but it can also just as easily lead to very bad and unanticipated behavior. For instance what if instead of making sure something wasn’t Nothing, the code made the argument Nothing?

<Extension()> _
Sub EvilMethod(ByRef p1 As Object)
    p1 = Nothing
End Sub

Sub Example2()
    Dim s As New Student()
    s.PrintClass()
    s.EvilMethod()
    s.PrintName()   ' Causes a NullReferenceException
End Sub

The code runs and throws an exception. A developer attaches a debugger, looks at the exception but immediately notes there are 2 other method calls just above. How could they succeed but the last throw a NullReferenceException’ Certainly this may be a fun joke but in production code a fellow developer would not be amused.

More importantly though the behavior when passing an extension method target by reference is flat out unexpected. Who expects what appears to an instance method call to be able to modify ‘Me/this’’ Probably not too many people. Coding is difficult enough without creating patterns that will blow away anyone who owns your code down the line. Code should do it’s best to be clear and produce predictable results.

Still, an interesting feature to toy with.

Immutable Collections And Backwards Compatibility

2008-12-10T00:00:00+00:00

When developing my immutable collections library, I spent a lot of time on usability. After all, if a library is not useful then what’s the point?

A big portion of usability is being able to work with existing frameworks and technologies. For .Net and collections that means items like Data binding, LINQ etc ‘ Without integrating into popular technologies the usefulness of a particular library is severely impacted and hence usability decreases.

Most of the existing collection based infrastructures use the .Net collection interfaces as their form of abstraction. The most straight forward way to ensure compatibility is to implement these interfaces on the collections in question. In particular [ICollection](http://msdn.microsoft.com/en-us/library/92t2ye13.aspx), [IList](http://msdn.microsoft.com/en-us/library/5y536ey6.aspx) and [IEnumerable](http://msdn.microsoft.com/en-us/library/9eekhta0.aspx) are the main abstractions. Lets investigate which ones an Immutable collection should be implementing in order to effectively integrate into existing collection based infrastructures.

**IEnumerable**

This is the easiest decision. IEnumerable represents a read only, one element at time view on a sequence of objects. Immutable collections can easily and reliably implement a IEnumerable. This is a no brainer. Implement.

**ICollection**

This interfaces represents a general collection class. Unfortunately this interface is meant to represent a mutable collection class and implements such methods as Add, Clear and Remove. These methods cannot be implemented on an Immutable collection given the current design. All three of these methods are void returning methods because the collection is meant to be changed in place. Immutable collections can support these operations but it involves returning a new instance of the collection.

public sealed class ImmutableCollection<T> : ICollection<T> {
    public ImmutableCollection<T> Add(T item) {
        // Actually add 
        // ...
    }

    #region ICollection<T> Members

    void ICollection<T>.Add(T item) {
        var created = this.Add(item);
        // What to do with created'
    }

    ...
}

But wait! The interface does support a property named IsReadOnly. The intention of this property is to allow an interface to programmatically declare they do not support modifications. A read only collection can just implement this interface, throw a NotSupportedException for all of the mutable methods and return true for IsReadOnly and presto we have a suitable interface for an immutable collection.

Or do we?

The design for ICollection with respect to read only or immutable collections is not optimal. It attempts to combine to separate behaviors into a single interface: mutable and readonly view of a collection. Dual purpose interfaces run into problems because it's impossible to separate out the behaviors at compile time. This is especially problematic when the behaviors are conflicting. There is no way a read only collection can prevent itself from being passed to a function that expects a mutable collection at compile time. Nor can a consumer who intends to mutate a collection prevent a read- only collection from being passed.

static void DisplayForEdit<T>(ICollection<T> col) {
    // ...
    m_clearButton.Click += (x, y) => col.Clear(); 
}

static void Example1() {
    ImmutableCollection<int> col = ImmutableCollection.Create(new int[] { 1, 2, 3, 4 });
    DisplayForEdit(col);    // Will fail as soon as Clear is clicked
}

But isn’t it the responsibility of the user of ICollection to verify that IsReadOnly is false before mutating a instance? Given the current design of ICollection it is indeed both the responsibility of the consumer to verify this and the implementer to ensure they are not called incorrectly. The problem with putting responsibility on the consumer though is they have to 1) know about read only uses of ICollection and 2) actually care about it.

A quick search of the BCL with reflector can give us evidence of how much consumers actually check for the read only scenario. For the search I used mscorlib, System, System.Xml, System.Data, System.Drawing and System.Core and System.Windows.Forms. And the number of classes which actually take into account ICollection.IsReadOnly is ' 1. System.Collections.ObjectModel.Collection. That's it.

So even if an immutable collection implements this interface in a read-only fashion there’s little chance anyone will be checking for it.

**IList**

IList inherits from ICollection and hence suffers from all of the same problems

Decision Time

In order to facilitate usability with existing frameworks immutable collections are forced to implement interfaces for which they have no possible way of implementing properly. If collections implement these interfaces they will be trading usability for compile time validation. Even worse is the conversion is implicit which prevents even basic searches for places this conversion occurs. This is a heavy price to pay for compatibility.

After debating this for awhile I decided that loss of compile time validation was a too heavy of a price to pay for the default scenario. But trading away usability was also unacceptable. As a compromise I opted for adding a compatibility layer to the collections. Instead of implementing the ICollection and IList collections directly I created a set of proxy objects that implement the interfaces on behalf of the immutable collections.

In order to centralize this effort I created a factory class, CollectionUtility, which contains appropriate overloads for all of my immutable collection classes ¹.

public static class CollectionUtility {
    public static IEnumerable<T> CreateEmptyEnumerable<T>();
    public static IEnumerable<T> CreateEnumerable<T>(T value);
    public static ICollection<T> CreateICollection<T>(IReadOnlyCollection<T> col);
    public static IDictionary<TKey, TValue> CreateIDictionary<TKey, TValue>(IReadOnlyMap<TKey, TValue> map);
    public static IList<T> CreateIList<T>(IReadOnlyList<T> list);
    public static ICollection CreateObjectICollection<T>(IReadOnlyCollection<T> col);
    public static IDictionary CreateObjectIDictionary<TKey, TValue>(IReadOnlyMap<TKey, TValue> map);
    public static IList CreateObjectIList<T>(IReadOnlyList<T> list);
    public static IEnumerable<int> GetRangeCount(int start, int count);
}

The proxy objects live as private inner classes inside CollectionUtility. They implement the collection interfaces in the most read-only way possible. When confronted with mutating calls, the proxies throw NotSupportedException.

So at the end of the day I have compile time validation for immutable collections. If a developer wants to use them in a less than safe scenario it requires an explicit conversion.

static void Example2() { 
  var col = ImmutableCollection.Create(new int[] { 1, 2, 3, 4 }); 
  // Still fails, but explicit conversion required 
  DisplayForEdit(CollectionUtility.CreateICollection(col)); 
}

I feel like this as an appropriate tradeoff. In the worst case scenario, a developer can search for all accesses of the CollectionUtility class and find places where a proxy is being created.

Next time, lets take a look at a different way of approaching an interface hierarchy for a set of collections. One that will allow us to avoid this problem altogether going forward.

Edit: Updated the exception to be NotSupportedException

It actually contains overloads for a set of truly read only collection interfaces that I wrote for my library but we’ll get to that another time. ↩

Mapping Linq To F

2008-12-02T00:00:00+00:00

In my projects with F#, I often find that I know exactly what type of sequence transformation I want to run, yet I spend all of my time trying to find it!!! The problem is I’m used to query comprehensions in LINQ terminology. Select, Where and SelectMany are so ingrained into my programming they are practically done through muscle memory.

Unfortunately there is a disconnect between the naming convention used in LINQ and the ones used in F#. Some of the names are more intuitive than others. I’ve now spent enough time digging through Seq.fs that it’s time to draw up a table that maps various LINQ functions to their equivalent method in F#.

If nothing else, it will be a good table for me to come back to. Below is a list of many methods in Enumerable, mapped to the equivalent function in F#’s seq

Aggregate ‘> fold
All ‘> for_all
Any ‘> exists
Average ‘> average
Cast ‘> cast
Count ‘> length
Distinct ‘> distinct
ElementAt ‘> nth
Empty ‘> empty
First ‘> hd
GroupBy ‘> group_by
Max ‘> max
Min ‘> min
OrderBy ‘> sort_by
Select ‘> map, mapi
SelectMany ‘> concat, map_concat
SequenceEqual ‘> compare
Skip ‘> skip
SkipWhile ‘> skip_while
Sum ‘> sum
Where ‘> filter

Below are the functions for which an equivalent was not found (or was quite simply overlooked). Most of these can be added with a combination of other simple transformations.

DefaultIfEmpty
Except
FirstOrDefault
GroupJoin
Intersect
Join
Last
LastOrDefault
LongCount
OfType
Reverse
Single
SingleOrDefault
Union

Sorting Out Binary Files

2008-12-01T00:00:00+00:00

I constantly get tripped up in my powershell scripts/commands because I run them against a binary file. In particular when I’m searching through a directory structure looking for a particular string or regex. I’ve found the simplest way to avoid this problem is to use a simple regex check to filter out the known binary files.

#==============================================================================
# Is this an extension for a binary file type? 
#==============================================================================
function Is-BinaryExtension() {
    param ( [string]$extension = $(throw "Need an extension" ) ) 

    $binRegex= "^(\.)?(lib|exe|obj|bin|tlb|pdb|doc|ncb|dll|pch)$"
    return $extension -match $binRegex
}

function Is-BinaryFileName() { 
    param ( [string]$fileName = $(throw "Need a file Name") )
    return (Is-BinaryExtension [IO.Path]::GetExtension($fileName))
}

Now running a recursive search on a directory structure is quick and easy

function Select-StringRecurse() {
    param ( [string]$text = $(throw "Need text to search for"),
            [string[]]$include = "*",
            [switch]$all= $false)

    gci -re -in $include | 
        ? { -not $_.PSIsContainer } | 
        ? { ($all) -or (-not (Is-BinaryExtension $_.Extension)) } |
        % { write-debug "Considering: $($_.FullName)"; ss $text $_.FullName }
}   

Comparing Continuations In C And F Part 3 Double Wrong

2008-11-13T00:00:00+00:00

Is it better to be wrong once or to be right then think you’re wrong but find out you were right but wrong about being wrong? Besides the obvious be right the first time, it’s certainly an educational experience.

Here’s the original sample:

let FoldRight combine (sequence:seq<'a>) acc = 
    use e = sequence.GetEnumerator()
    let rec inner cont = 
        match e.MoveNext() with
            | true -> 
                let cur = e.Current
                inner (fun racc -> cont (combine cur racc))
            | false -> cont acc
    inner (fun x -> x )

Brian McNamara pointed out I wasn’t considering all of the call sites for this sample. In addition to the recursive call to ‘inner’ and the initial inner call, there is the actual recursive invocation of the of the continuations. Effectively the ‘inner’ function is building up a list of list of lambdas which call the combine function. The output of the combine function is simply passed into the next lambda in the list. The last lambda in the list is the identity lambda and returns the final call to combine. This value is the actual value returned from the initial invocation ‘cont acc’. Lambdas are methods under the hood. Without a tail instruction, this chain of lambda calls will just as easily overflow the stack.

Digging deeper into the compiled F# code we can view this call and indeed it is done with tail recursion.

.method public virtual instance !V Invoke(!U racc) cil managed
{
    .maxstack 8
    L_0000: nop 
    L_0001: ldarg.0 
    L_0002: ldfld class [FSharp.Core]Microsoft.FSharp.Core.FastFunc`2 Program/clo@9::cont
    L_0007: ldarg.0 
    L_0008: ldfld class [FSharp.Core]Microsoft.FSharp.Core.FastFunc`2> Program/clo@9::combine
    L_000d: ldarg.0 
    L_000e: ldfld !2 Program/clo@9::cur
    L_0013: ldarg.1 
    L_0014: call !!0 [FSharp.Core]Microsoft.FSharp.Core.FastFunc`2::InvokeFast2(class [FSharp.Core]Microsoft.FSharp.Core.FastFunc`2>, !0, !1)
    L_0019: tail 
    L_001b: callvirt instance !1 [FSharp.Core]Microsoft.FSharp.Core.FastFunc`2::Invoke(!0)
    L_0020: ret 
}

The below code more accurately resembles the equivalent C# code that is generated for the above F# sample?? (thanks Brian!).

public static TAcc Inner<TSource, TAcc>(
    this IEnumerator<TSource> e,
    Func<TAcc, TSource, TAcc> combine,
    TAcc start,
    Func<TAcc, TAcc> cont)
{
    while (e.MoveNext())
    {
        var cur = e.Current;
        Func<TAcc, TAcc> innerCont = cont;
        cont = (x) => /*need .tail here */innerCont(combine(x, cur));
    }
    return cont(start);
}

public static TAcc FoldRight<TSource, TAcc>(
    this IEnumerable<TSource> enumerable,
    Func<TAcc, TSource, TAcc> combine,
    TAcc start)
{
    using (var e = enumerable.GetEnumerator())
    {
        return Inner(e, combine, start, (x) => x);
    }
}

Previous Entries

Properly Incrementing An Intptr

2008-11-11T00:00:00+00:00

Just as native pointer types are moved around with pointer arithmetic in native code, it can also be useful to move IntPtr types around in managed code. Say for instance there is an IntPtr available which points to a native array of Dog instances. To access the values of that array individually requires pointer arithmetic of a fashion.

For the most part this is a straight forward operation if the underlying native memory is understood. Say there is an array of Dog instances with length 10 and the Dog structure has a size of 8. The total amount of memory will be 80 bytes and with a valid dog instance being available at every 8 bytes. So if the start address is 1000 then 1000,1008,1016 and so on will point to a valid instance.

The native size of any data structure can be calculated via Marshal.SizeOf(tyepof(Dog)) ¹. With a pointer to the start of the array, the Nth Dog instance can be accessed with a pointer of address = (N*sizeof(Dog))+startAddress

The address of a pointer can be accessed by 1 of 2 functions

.ToIn32()
.ToInt64()

Unless you are writing an application that will every only run on a 32 bit system, don’t use method #1 (even then still don’t). Native pointer addresses vary in size depending on version of the OS a program is running on. 64 bit systems have a much larger address size (long vs int). Consequently calling .ToInt32 on a 64bit system will truncate the actual address to a valid that is no longer valid. This will eventually lead to a random error PInvoke’ing a function that is difficult to track down.

Instead use .ToInt64(). This method is safe on both 32 and 64 bit systems. Additionally constructing an IntPtr instance with either value is safe. The class knows what version of windows it’s executing on and will adjust the size in a safe way ².

In many of my projects I define a class similar to the following to take care of this automatically.

public static class IntPtrExtensions
{
    public static IntPtr Increment(this IntPtr ptr, int cbSize)
    {
        return new IntPtr(ptr.ToInt64() + cbSize);
    }

    public static IntPtr Increment<T>(this IntPtr ptr)
    {
        return ptr.Increment(Marshal.SizeOf(typeof(T)));
    }

    public static T ElementAt<T>(this IntPtr ptr, int index)
    {
        var offset = Marshal.SizeOf(typeof(T))*index;
        var offsetPtr = ptr.Increment(offset);
        return (T)Marshal.PtrToStructure(offsetPtr, typeof(T));
    }
}

It’s highly advisable to not calculate this value yourself. ↩
See the post ‘Is IntPtr(long) truncating?’ for more details ↩

Comparing Continuations In C And F Part 2

2008-11-11T00:00:00+00:00

In my last post I went over the differences between using a continuation in F# and C#. As it turns out I was right about the limits and symptoms but wrong about the reason.

The F# code does indeed generate tail calls for part of the continuation. However this is only a very small portion of the actual code and is in fact only generated for the call in the empty case. I misread this function to be the call for the overall continuation. Instead it is the function for the entire ‘inner’ lambda.

So why does F# perform differently than C# in this scenario?

Andrew Kennedy pointed out that F# will actually transform the ‘inner’ function into a loop. In affect the code generated looks like the following.

TypeFunc func = this._self3;
while (true)
{
    if (!this.e.MoveNext())
    {
        break;
    }
    A cur = this.e.Current;
    cont = new Program.clo@9<U V, A ,>(this.combine, cont, cur);
}
return cont.Invoke(this.acc);

The actual transformation into a loop is what is preventing F# from overflowing the stack here. Iteration incurs no stack overhead in this case.

Even more interesting is that the tail opcode is quite simply ignored when dealing with un-trusted code. It therefore cannot be relied on to generate performant code in all scenarios.

Comparing Continuations In F And C

2008-11-10T00:00:00+00:00

Lately I’ve been playing quite a bit with F#. I have several hobby projects I’m working on that take up a bit of my time. But when I’m not playing around with F# I’m exploring ways to apply certain functional patterns to actual coding on the job and/or porting to my functional library: RantPack.

Recently I’ve been playing around with continuations in F#. I thought this was a great topic to do a F# comparison with other languages. In this case C#.

Let’s examine a classic use of continuations: a right fold on a list. For a detailed explanation of fold right and the use of a continuation I suggest taking a look at Brian’s discussion. If you’re unfamiliar with continuations I highly suggest that you take a look at this post as Brian gives a great breakdown of continuations and their uses.

Here’s a quick refresher on continuations by example. Fold right is an operation which reduces a sequence of elements into a single element by processing the list from right to left. It’s similar to the LINQ Aggregate method except Aggregate operations left to right.

For this post we’ll be writing FoldRight against a sequence. I chose sequence vs. the traditional list because it’s present in both languages (F# = seq<’a>, C# = IEnumerable). It is possible to other F# data structures in C# but the comparison is cleaner when using a type that is native to both languages.

Sequences are a left to right data structure so processing it right to left is not natural. After all, with a sequence all the developer has is current and whether or not there is a next element. Processing the list in a right to left fashion can be done by such acts as reversing the list, or doing a head recursive call. Both have their detractors.

Continuations are a different way to process the list. Instead of processing the list directly we process the list a single element at a time building up a continuation along the way. For each element a lambda expression is generated representing the work needed to be done for that element. The value calculated within the lambda will then be passed to the lambda calculated for the previous element. Once we hit the end of the list, we essentially have a chain of lambda expressions which process each element in the list in reverse order. All that is needed is to call the final lambda with the starting value and we will effectively process the list in reverse order.

Simple enough’ Lets take a look at the code.

F# Code

let FoldRight combine (sequence:seq<'a>) acc = 
    use e = sequence.GetEnumerator()
    let rec inner cont = 
        match e.MoveNext() with
            | true -> 
                let cur = e.Current
                inner (fun racc -> cont (combine cur racc))
            | false -> cont acc
    inner (fun x -> x )

C# Code??

public static TAcc FoldRight<TSource, TAcc>(
    this IEnumerable<TSource> enumerable, 
    Func<TAcc, TSource, TAcc> combine, 
    TAcc start) {

    using (var e = enumerable.GetEnumerator()) {
        Func<Func<TAcc, TAcc>, TAcc> inner = null;
        inner = (cont) => {
              if (e.MoveNext()) {
                  var cur = e.Current;
                  Func<TAcc, TAcc> innerCont = (x) => cont(combine(x, cur));
                  return inner(innerCont);
              } else {
                  return cont(start);
              }
          };
        return inner(x => x);
    }
}

My immediate reaction to the two samples is the conciseness of the F# code. This is a not a criticism of C# though. F# is designed to be a concise language and it’s delivery on that goal is evident in this sample.

What makes the big difference here is the type inference power of F#. In the C# sample there are 6 explicit types listed in the code sample. The F# sample only has a single type listed. The compiler is able to infer and/or generate the rest of the signatures. F# also requires less explicit generic parameters 1 vs. 2 in C#.

The next big difference I see is the awkward way in which the inner lambda expression must be declared in C#. The lambda expression is called recursively in order to setup the continuation. In order to do that in C# the lambda must be declared and defined in separate expressions. Otherwise, a self reference of ‘inner’ inside the body of ‘inner’ will generate a used before defined warning from the compiler.

The IL

Examining the full IL of both functions would take several blog posts. Not to mention that trying to read disassembled F# much less IL, is like trying to read disassembled C++. An interesting exercise but a bit time consuming.

I did want to focus a bit on one portion of the generated IL. There is a very significant difference in the way the recursive call to the ‘inner’ lambda is made.

    L_0032: ldarg.1 
    L_0033: ldarg.0 
    L_0034: ldfld !0 Test/clo@6T::acc
    L_0039: tail 
    L_003b: callvirt instance !1 [FSharp.Core]Microsoft.FSharp.Core.FastFunc`2::Invoke(!0)

    L_0077: ldarg.0 
    L_0078: ldfld class [System.Core]System.Func`2, !1> ConsoleApplication1.Extensions/<>c__DisplayClass8::inner
    L_007d: ldloc.0 
    L_007e: callvirt instance !1 [System.Core]System.Func`2, !TAcc>::Invoke(!0)
    L_0083: stloc.3 

In both cases the first 3 lines are building up the 2 parameters necessary for the recursive lambda call. The closures are structured somewhat differently but the same basic operation is being done.

The key difference between the languages is F#’s use of the tail opcode. This opcode tells the CLR the to call the next method in a tail recursive fashion. This causes the CLR to remove the current method frame from the stack before the next method is called. Because the method is removed from the stack, the recursive call takes up no additional stack space. This is true no matter how many times the function is called.

The C# IL does not have this opcode. So the recursive call will happen with the current method on the frame. With a big enough sequence this will cause the process to run out of stack space and generate a StackOverflowException. This creates a limitation on the number of elements the C# sample can process.

Limits

I explored the limits of both samples on my home laptop. I generated a simple example to sum the sequence with the fold right. Note: For a sum of ints, a fold left is just as good, but it serves fine for this sample.

let sum = FoldRight (fun x y -> x + y) [1..1000000] 0
printfn "%d" sum

var source = Enumerable.Range(1, 9397);
var result = source.FoldRight((x, y) => x + y, 0);
Console.WriteLine("{0}", result);

The C# sample can process a maximum size of 9397 elements. After that I encounter a stackoverflow exception. The F# sample however can easily process 1,000,000 elements.

Closing note. This not meant to be a post criticizing C#. It’s meant to be a general comparison of the same technique in two managed languages. This is a scenario that is far less likely to occur in a C# program. In an F# program it’s quite simply an expectation and hence the F# compiler is optimized for this scenario.

Dereference A Double Intptr

2008-11-05T00:00:00+00:00

A common PInvoke question is how to deal with a double pointer. More specifically, how can one dereference an IntPtr to another pointer without using unsafe code?

Dereferencing a double pointer is done the same way a dereference to any other structure is done: Marshal.PtrToStructure. PtrToStructure is used to transform a native pointer, in the form of an IntPtr, into a managed version of the native data structure the native pointer points to.

In the case of a double pointer, the native data structure the pointer points to is just another native pointer. The managed equivalent is the IntPtr (or UIntPtr) class.

For Example, say we had the following native data signature

void GetDoublePointer(int** ppData)

This function returns a pointer to a pointer that points to an int. (Pointer->Pointer->int). We can then use the following C# code to access the final int value.

[DllImport("PInvokeSample.dll")]
static extern void GetDoublePointer(IntPtr doublePtr);

static void Main(string[] args)
{
    var ptr = Marshal.AllocHGlobal(Marshal.SizeOf(typeof(IntPtr)));
    try
    {
        GetDoublePointer(ptr);
        var deref1 = (IntPtr)Marshal.PtrToStructure(ptr, typeof(IntPtr));
        var deref2 = (int)Marshal.PtrToStructure(deref1, typeof(int));
        Console.WriteLine(deref2);
    }
    finally
    {
        Marshal.FreeHGlobal(ptr);
    }
}

Pinvoke And Com Objects

2008-11-04T00:00:00+00:00

I’ve occasionally found the need to mix COM interop and PInvoke. For certain scenarios it’s just easier to code up a PInvoke declaration and function. It’s perfectly legal to include COM objects in these scenarios provided the appropriate Marshal attributes are added to the signature.

The easiest way to accomplish scenario is to have the native signature only expose IUnknown instances. On the managed side use an object declaration annotated with MarshalAs(UnmanagedType.IUnknown). Example:

[DllImport("SomeDll.dll")]
[return: MarshalAs(UnmanagedType.IUnknown)]
public static extern object GetSomeComObject();

One item to remember though is how to managed the ref counting in this scenario. In any case where a COM object is considered to be coming out of the PInvoke signature, the CLR will assume that it has an obligation to call IUnknown::Release() at some point in the future. The corresponding native code must take this into account and appropriately AddRef() the object.

This includes any scenario, as displayed above, where the COM object is the actual return value of the function ¹.

Got bit by this last week. ↩

Stop The Beeping

2008-11-03T00:00:00+00:00

Recently I was working on a PowerShell script which involved translating byte arrays into strings using the appropriate encoding. Unfortunately I kept getting the wrong choice for encoding and printed out essentially random data to the console screen.

Unfortunately random data + windows console screens = beeps, beeps and lots more beeps.

What’s even worse is these beeps queue up. Almost a full minute after a given encoding mistake I was still paying the price. Highly annoying.

There is no way to disable the beeps on an individual console level. But it can be disabled on a system wide level. Beep is a windows service so it just needs to be stopped

net stop Beep

This does need to be executed with Admin privileges. After that I can code beep free.

Presentation Follow Up

2008-10-29T00:00:00+00:00

Thanks to everyone who showed up at the presentation on monday. For those interested, I’ve uploaded the contents of the presentation here.

I was unable to upload the large DB file used during the demo due to size limitations (sky drive will only let me upload a 50 MB file). But better than wasting both mine and your bandwidth, I included the script used to generate the db file.

The file is a powershell script named GenerateDb.ps1. If you pass the -argLarge switch, it will generate the 120MB demo file.

PS>.\GenerateDb.ps1 -argLarge

Is Intptr Long Truncating

2008-10-28T00:00:00+00:00

The short answer is: No, not when it matters

A colleague and I were discussing a particular scenario around IntPtr,PInvoke and 64 bit correctness. Eventually our discussion lead us to the IntPtr constructor which takes a long. To my surprise the code for the constructor is the following.

public unsafe IntPtr(long value) { this.m_value = (void*) ((int) value); }

The problem is long value is arbitrarily truncated to an int. This has the effect of essentially losing any address over the 4 GB range (in other words, no 64 bit addresses). This much to big of a hole to actually be the real behavior so I decided to see if it was a bug in the disassembler. I was using .Net Reflector so I switched to IL mode.

    L_0000: ldarg.0 
    L_0001: ldarg.1 
    L_0002: conv.ovf.i4 
    L_0003: conv.i 
    L_0004: stfld void* System.IntPtr::m_value
    L_0009: ret 

This confirmed it is indeed truncating the value (and doing an overflow check to boot). But wait, mscorlib.dll is a processor specific DLL so perhaps this is just a 32 bit OS thing. I switched over to a 64 bit machine, fired up Reflctor and found to my dismay that it had the exact same code.

After a few minutes I thought to open up task manager and to my surprise reflector was running in a WoW64 bit process. This meant it was still loading up the 32 bit version of mscorlib.dll. Next I fired up ildasm, loaded up a 64 bit mscorlib and confirmed that the code will not truncate on 64 bit machines.

  IL_0000:  ldarg.0
  IL_0001:  ldarg.1
  IL_0002:  conv.u
  IL_0003:  stfld      void* System.IntPtr::m_value
  IL_0008:  ret

The conv.u code is a conversion to unsigned native platform int. On a 64 bit machine this will be an unsigned 8 byte number(see OpCodes.Conv_U for more details).

So what does this mean for the developer. Essentially IntPtr(long) will do the right thing independently of the platform a developer is using. On a 32 bit platform it will (correctly) throw exceptions if a non-4GB address is passed in. In 64 bit land it will essentially do nothing and rely on the programmer to give correct addresses.

Properly Handling A Winforms Timer Event

2008-10-27T00:00:00+00:00

The WinForms Timer class allows the user to perform a particular action at a set interval. Timer objects fire a Tick event at the set time which users can easily respond to. This is very useful if a developer wants to check for a particular condition say every 2 seconds ( for the remainder of this article I’m going to use 2 seconds as a practical example even though it’s really any arbitrary time period).

Occasionally users are surprised to find that the Tick event will fire much faster than they are expecting. Instead of waiting for 2 seconds between calls, they event will fire almost immediately after one is finished processing.

What’s going on here is a side effect of how this event works under the hood. The interval for the timer event is calculated in real world time. So quite literally every 2 seconds Windows will consider the internal reached and will issue a new tick message. The next time a WinForms event is not executing developer code a tick event is raised ¹.

So imagine we had the following code.

Private Sub OnTimerTick() Handles m_timer.Tick
    RunSomeOperation()
End Sub

Consider what happens if RunSomeOperation takes longer than 2 seconds. The Tick event is fired in real time so while we’re in the middle of RunSomeOperation, another Tick event is being queued up for processing. As soon as we leave OnTimerTick we’re back in WinForms code which sees a Tick event and promptly raises it which puts us right back in OnTimerTick.

This is contrary to what most people expect. Most people expect the Tick event to fire 2 seconds after their code is finished executing.

To work around this developers should stop the timer when processing a timer event. Just before exiting the event handler, re-enable the timer. This will cause Windows to start calculating the interval from the start. This has the effect of making the timer event fire 2 seconds after developer code stops executing.

    Private Sub OnTimerTick() Handles m_timer.Tick
        m_timer.Stop()
        Try
            RunSomeOperation()
        Finally
            m_timer.Start()
        End Try
    End Sub

This is not 100% true. It’s really whenever the Application begins to pump messages again. Message pumping, more specifically when it does and does not occur, is too involved for this discussion. ↩

Redefining Success

2008-10-24T00:00:00+00:00

Spent about an hour debugging a bit of code today. I was attempting to read data from a particular source and kept getting back failure codes. After some debugging I discovered the data didn’t actually exist in the source I was reading from.

This put me back to investigating where I wrote the data out. Restarted the scenario and verified that I actually called the data writing API and that it succeeded.

Now what’ Well the data clearly wasn’t there so I concluded the data writing must be failing in some odd way. I eventually found the data writing code and was horrified to find the following definition.

HRESULT WriteSomeData(...) {
  // We don't support data of this type
  return S_OK;
}

Personally I thought this warranted an error code (perhaps E_NOTIMPL). But given the situation I must conclude the author successfully failed to write the data.

Presenting At Net Developers Association Meeting Oct 27

2008-10-23T00:00:00+00:00

I will be presenting at the Net Developers Associating meeting this upcoming Monday. If you’re interested in hearing me ramble on in person please drop by.

Event Details:

When: Oct 27 7:00PM
Where: Microsoft Building 40: The Steptoe Room (Map)
Who: Me
Why: Blogging’s fun but it’s great to go out and talk to people in person

My talk will center around how to get the most out of a LINQ query. I’ll be covering a lot of the misunderstood aspects of LINQ such as deferred execution. Most bit of the talk will focus on the MSDN article I wrote a few months back.

When Can You Catch A Stackoverflowexception

2008-10-22T00:00:00+00:00

Answer: When you’re the one who threw it.

Starting with the CLR version 2.0, the policy for handling a StackOverflowException was changed. User code can no longer handle the exception¹. Instead the CLR will simply terminate the process.

This is not 100% true though. User code can still handle StackOverflowExceptions which are artificially thrown. That is thrown by the user instead of resulting from an actual overflow of the stack. This is in contradiction to the documentation but can be demonstrated with a quick and dirty sample program (see end of the post).

This is a trivial point for sure. Yet I feel the need to point it out because I recently saw a newsgroup conversation where someone posted sample exception logging code and happened to use a StackOverflowException in the sample. Their sample explicitly threw the exception so it worked and they had good reason to suspect it worked in production as well. I was equally amazed it worked at all.

Please don’t take this post as advocating that you should handle a StackOverflowException (you shouldn’t). This is merely an oddity I found interesting. Personally I’d prefer it not be catch-able in any circumstance.

public static void CatchStackOverflow1() {
    try {
        throw new StackOverflowException();
    } catch (StackOverflowException ex) {
        // Executes and handles the exception.  User code continues
        Console.WriteLine(ex.Message);
    }
}

static int CreateRealOverflow(int p1) {
    return 42 + CreateRealOverflow(p1 + 1);
}

public static void CatchStackOverflow2() {
    try {
        CreateRealOverflow(42);
    } catch (StackOverflowException ex) {
        // Will not execute
        Console.WriteLine(ex.Message);
    }
}

static void Main(string[] args) {
    CatchStackOverflow1();
    CatchStackOverflow2();
}

Unless you are hosting the CLR in which case you can implement some recovery mechanism. This is certainly the exception though and not the rule. ↩

Program Files I Just Want The 32 Bit Version

2008-10-21T00:00:00+00:00

As part of my transition into using 64 bit windows I keep running into a problem with some scripts.

I have a whole set of Powershell scripts that are dedicated to ensuring certain programs are installed on all of my dev machines. Or that certain customizations are needed. A lot of these do file existence checks inside of Program Files.

Unfortunately in 64 bit windows there are actually two Program Files folders. One for 64 bit programs and a separate one for 32 bit programs that operate in Wow64 mode. All code which uses $env:ProgramFiles will point to the 64 bit version. Most of the programs I custom install (i.e. gvim are actually 32 bit programs).

Getting the 32 bit Program Files directory is simple enough: ${env:ProgramFiles(x86)}. Yet it’s not portable back to a 32 bit version of windows. Yet another function to the rescue

function Get-ProgramFiles32() {
    if (Test-Win64 ) {
        return ${env:ProgramFiles(x86)}
    }
    
    return $env:ProgramFiles
}

Custom Exceptions When Should You Create Them

2008-10-20T00:00:00+00:00

I think the best answer is: rarely.’? It’s really hard to go straight to a justification here though. I find that answering a different question will eventually shed led on when to create a new exception.

“What are the benefits of creating a new/custom exception?”

The answers I come up with or have heard before are …

Provides a type safe mechanism for a developer to detect an error condition.

Without a custom exception a developer would be forced to compare an existing value on an existing exception class in order to determine if a particular error occurred. To ensure this was unique across applications and environments a unique identifier would need to be inserted as well (likely a GUID). The resulting code would be quite ugly and maintainability would be a question mark.

Add situation specific data to an exception

Ideally this data would help another developer track down the source of the error.

But an entirely new exception is not needed to achieve this goal. The base Exception class contains a property named Data which is an open dictionary. Any code which raises an exception is free to add custom data into the exception. This data can then be accessed by the handling code. Unique values still need to be dealt with. But in my experience, accessing a GUID based custom property is a bit more accepted than catching an exception based on some GUID property.

This is especially true now with extension methods. The lack of type safety in the dictionary and potentially constant string sharing for property names could be hidden inside an extension method

No existing exception adequately described my problem

I’d argue against this because of InvalidOperationException. Invalid operation should cover most exceptional cases

So of these reasons only #1 is a stand alone benefit of a new exception. Now lets re-state this advantage in the context of the original question.

Creating a new exception allows a developer to catch them

Simple right’ But what benefit does this give to a developer’ From my experience there are only two reasons to catch an exception.

It’s a particular for which the underlying exceptional problem can be corrected. Perhaps the user is prompted to fix the condition by performing some outside action (like re-inserting a disk).
Logging errors for post-mortem debugging. Note: You could also use the Data dictionary and extension methods above to introduce a new method ShouldLog(). This would allow you to avoid creating a new exception type.

Now I think we can answer the original question of this blog post.

You should only create a new exception if you expect developers to take corrective action for the problem or to log for post mortem debugging.

Otherwise, what benefit does creating this exception give you?

Community Interview

2008-10-17T00:00:00+00:00

Recently I did a community interview with Microsoft MVP Alessandro Del Sole.

The Italian version can be read here.?? And no, I don’t speak Italian. The original transcript is below.

1. Nice to meet you Jared Parsons! Let’s begin by asking a simple question: where are you from?

Born in Kentucky, raised in Tennessee, lived in Alabama for awhile, then back to Tennessee and went to college in Georgia. Usually I simplify it by saying I grew up all over the southeast.

2. We all know that you’re a Visual Basic Team member, but could we know something more about your role inside the Team?

I’m a developer in the VB team. I started out on the VB Compiler several years ago. Towards the end of Visual Studio 2008 I switched to the VB expression evaluator (aka debugger, watch, immediate window). This role expanded to other parts of the VB IDE experience including a lot of features users will be (hopefully) really excited about for the next release of Visual Studio.

3. What’s your favorite Visual Basic feature regarding the language and/or the IDE?

My favorite VB feature is usually whatever I’m implementing at the moment I’m asked this questionJ. Luckily I’m in between features at the moment so I can give an unbiased answer. XML Literals are my favorite VB feature. It really allows me to seamlessly integrate my data and queries into my application. I feel like I find a new way to use it every day.

4. How long have you been working for Microsoft and what was your occupation before joining the VB Team?

I’ve been working at Microsoft for 4 ?? years now. I joined Microsoft immediately after college in the in the Visual Studio Team System division. More specifically I was a developer on Team Developer and Team Architect. After shipping Visual Studio 2005 I was eager for new challenges. I spent a good deal of my time in college working on low level systems and being a teaching assistant for an introduction to compiler courses. I was interested in getting back to these roots. A job opened up on the VB compiler team and it was the perfect fit.

5. Why do you think VB is such a great language?

We have a tie here: flexibility and IDE. VB is an amazingly flexible language. It lends itself to whatever task I’m trying to do. On one end of the spectrum I have XML literals to work with my data and at the other end generics and lambda expressions to really geek out on a problem. The IDE support is also very good (I promise I’m not biased here). It’s really difficult to express everything the IDE offers to the programmer. The best way to understand though is to get out notepad and try and code up a VB application. That will really give you a handle on what the IDE is doing for you.

6. Have you ever been to Italy?

No but it’s definitely on the list of places to visit.

7. Would you like to tell us something about your life outside the office (how do you like to spend your free time, what do you like to do when/if you go out in the evening or when you’re at home and so on)?

Outside of work I still lead a largely geek lifestyle; lots of programming, blogging and book reading. I do manage to leave my geek role by playing a variety of sports. Right now I’m playing soccer (indoor and outdoor), softball and football. I’m not very picky when it comes to sports. As long as it’s athletic and preferably outdoors I’m there. It’s a great way to get release after a day of programming.

Of course there is the occasional Rock Band party as well.

8. Who is your favorite musician?

Much to my parent’s disappointment, I don’t have an ear for music (read that as 100% tone deaf). As a consequence my taste for music is ‘ bad according to my friends. I prefer to think of it as blissful ignorance. On that note (pun intentional) I love a good punk album or some techno while coding.

9. What’s your favorite food?

The hands down winner here is French Fries.

Powershell And 64 Bit Windows Helper Functions

2008-10-16T00:00:00+00:00

Recently at work I started using Windows 2008 64 bit edition. Mainly for hyper-v but powershell also comes as part of the deal.

I’m starting to work through the fun issues of getting some of my environment specific scripts to run in a 64 bit powershell process. The following scripts are turning out to be fairly handy

# Is this a Wow64 powershell host
function Test-Wow64() {
    return (Test-Win32) -and (test-path env:\PROCESSOR_ARCHITEW6432)
}

# Is this a 64 bit process
function Test-Win64() {
    return [IntPtr]::size -eq 8
}

# Is this a 32 bit process
function Test-Win32() {
    return [IntPtr]::size -eq 4
}

Regular Expression Limitations

2008-10-15T00:00:00+00:00

I’m struggling with an introduction line to this blog post so I’m just going to go for bluntness:?? Regular expressions are limited and it’s important to understand these limitations. Ok, now that the premise is out of the way, lets go to a sample.

For a time in college I was a teaching assistant in a introduction to Compiler course ¹. Regex’s played a large role in the class and inevitably a question or two would appear on a test and the final. My favorite question was the following

Imagine the C block comment token (/* /) was allowed to be recursive. For example / /* this is a comment */ */. Write a regex which will match all C comments.

We’d get loads of interesting answers but there is only one correct one. It’s not possible.

Why’ The keyword in this question is “recursive.”?? I prefer to call it “counting” though. Regular expressions a class of Finite Automata (FA). One of the properties of FA is they cannot be asked to match anything that is recursive (aka has the equal number of items on left hand side and right hand side). They lack the sufficient state necessary to do so. To accurately match this a Context Free Grammar is required. Enough with the terminology though.

Critics: Wait, you think asking a trick question on a test is a good idea?

Answer: At first no. I didn’t like this question when I encountered it as a student and it took me a precious 10 minutes to answer. I knew the answer right off the bat but I was hesitant to write it down. As a TA I liked this question slightly more. Namely because we drilled on this point from day 1 and most students would still miss the question ².

I didn’t truly appreciate this question until I started working professionally. I often see people struggling to create a functioning regex for patterns that it cannot possibly work for. One of the first places I saw it was in a piece of code that QA was (naturally) having a field day with. I was brought it for a code review and after seeing the regex and the problem I pointed out that it wouldn’t be possible. It took a few minutes but I was able to convince them a regex wouldn’t work. Up until that point no one had considered there wasn’t a regex solution, just that the one they had wasn’t good enough. Days were wasted in this venture.

Part of being a good programmer is recognizing the strengths and weakness’s of your tools. Most importantly though is knowing the limitations. It will save you hours and potentially days of wasted work.

As an exercise which of the following can a regex help with (answer at bottom under ³)?

A C# string
A C# string literal
An XML element
An XML attribute
A file system path
A C/VB/C# math expression
Email address
Matching a valid regex

As a side note, there are several regex engines out there that support the notion of recursion. These will allow you to match these types of patterns but are not technically regexs in the strictest since of the word. These are not strictly speaking regular expressions in computing theory but recursive extensions are quickly becoming a part of many libraries.

In a future (hopefully soon) post I will explore the recursive extensions to the .Net regular expression language.

It had various names CS2330 is the most infamous ↩
Even though we would go over it verbatim in class ↩
Regex will work for 1,2,4,5,7 ↩

PInvoke and bool or should I say BOOL?

2008-10-14T00:00:00+00:00

Boolean value types are a constant source of problems for people attempting to generate PInvoke signatures. It’s yet another case of managed and native types differing on size.

There are two common boolean types in native code: bool and BOOL. As with most PInvoke issue the main issue is to understand the size

bool: 1 byte
- bool SomeOperation();
BOOL: 4 bytes
- BOOL SomeOtherOperation();

Fortunately there is only one managed boolean type. Unfortunately though, it has 2 sizes that must be considered. In managed code a bool is a single byte. Yet the default marshalling for a bool will expand it to 4 bytes. So even though bool is 1 byte in managed code, it is by default compatible with the 4 byte native version BOOL.

Translating a PInvoke signature for a native 1 byte bool is simple enough though. The UnmanagedType.I1 value allows for this scenario. It tells the CLR that while the managed code has a boolean value the corresponding native item is the 1 byte version of bool. The CLR will then make the necessary size adjustments. When converting native -> managed it will consider a non-zero value true and 0 to be false.

Native:

bool SomeOperation();

Managed:

[return: MarshalAs(UnmanagedType.I1)]  
public static extern bool SomeOperation();

BOOL’s can be usefully marshalled in two different ways. The first is simply as a int (signed or unsigned). This works because int matches the size requirement of a BOOL and the sign is not really important as true or false is simply 0 or not 0.

public static extern int SomeOtherOperation();

This is certainly sub-optimal since we’re now returning a non-boolean type for what is intended to be a boolean operation but some people choose to Marshal it this way.

Marshaling a native BOOL to a managed bool works without any annotations. However it is preferred that you include the annotations to make it much clearer what the native data type is. The UnmanagedType enumeration contains a Bool value for just this purpose. This can be applied to any managed bool return or parameter.

[return: MarshalAs(UnmanagedType.Bool)]  
public static extern bool SomeOtherOperation();

And presto, we have maintained the boolean semantics of the operation.

Hint: If you are using the PInvoke Interop Assistant, it will automatically do these conversions for you.

Edit

Fixed a mixup on the default native size of bool. Thanks to Jachym Kouba for pointing out the mistake

If it's not broken maybe you should fix it anyway

2008-10-13T00:00:00+00:00

I know this is goes against conventional wisdom but it’s something I believe in. Every sufficiently large project has that section of code nobody wants to go near. The mere mention of it causes people to leave the room. It usually has a couple of properties

Old and Stagnant
Critical or important routine
Fragile
Written by a developer who doesn’t work there anymore
Undocumented
Untested
Poorly understood

But it works and usually has for several releases. By conventional wisdom it is considered “unbroken” and not worthy of changing.

I disagree. Very often I find towards the end of the cycle more and more bugs start popping up in this code. It’s a pattern I’ve seen in many groups across releases. These bugs are extremely expensive to fix because they are a) at the end of the cycle and b) being make in a piece of code that no one understands. Since it’s not broken nobody bothered with the code and hence it stayed poorly understood (and likely just as untested). The bugs are doubly worse because no one wants to go near that code for fear of breaking the world.

Why let this code hang around in it’s current state’ It’s bad and no one understands why. It’s a constant source of pain late in the cycle and often in the middle.

I prefer to take a more proactive approach; fix it.

At the end of a product cycle I like to identify these pieces of code and attack them in-between releases. This is a relatively safe point in the release cycle to change poorly understood code. There is an entire release cycle ahead to catch the bugs introduced (and there will be some).

I don’t break this code without thought. I prefer to take a pragmatic approach to the code in the following steps.

Attempt to understand the intent of the code (easier said than done). I contact the original author if possible
Poll the more senior developers on their understanding of what the code does
Design a new architecture if needed
Design a test plan (this is the absolute most important step. Why fix this if you don’t bother testing it?)
Make the change

I’ve done this, or overseen someone else, on several pieces of code over the last few years. Thus far it’s been a very big success.

Stackoverflow A New Addiction

2008-10-10T00:00:00+00:00

If you haven’t visited stackoverflow.com yet I encourage you to take a trip. It’s a great tag based newsgroup system. Check it out.

Vb Catch When Why So Special

2008-10-09T00:00:00+00:00

The VB Catch syntax has a particular feature not present in C#: When. It allows users to filter expressions based on something other than their type. Any arbitrary code can enter a When block to decide whether or not to handle an Exception

Sub Sub1()
    Try
        DoSomeAction()
    Catch ex As Exception When Filter(ex)
        Stop
    End Try
End Sub

Newsgroups often ask, “Why’s this so special? I could effectively get the same behavior out of C# by doing the following.”

static void Sub1()
{
    try
    {
        DoSomeAction();
    }
    catch (Exception ex)
    {
        if (Filter(ex))
        {
            throw;
        }
        HandleException();
    }
}

This is true to an extent. In both cases the code is handling an exception and making a decision, via calling Filter, as to whether or not to handle the exception. The subtle difference is when the Filter method is called.

In VB the When statement is actually implemented as an IL exception filter. When an exception is thrown, exception filters are processed before the stack is unwound. This means that if the Filter method created an error report that included the current stack trace, it would show the frame in which the exception occurred.

For example, in the code above if DoSomeAction() threw and the stack was examined in the Filter expression, the following stack would show up.

Notice how the DoSomeAction method is clearly visible? This is incredibly powerful for features like error reporting and investigation. It also allows you to set powerful breakpoints where the exact state of the error can be examined and not just the post mortem.

Alternatively, code executed in the C# block will occur after the stack is unwound. This gets rid of the culprit. As long as your not in optimized code you can usually use the stack trace properties to get the source of the exception but you won’t be able to examine the live state of the error.

Functional C#&58; Providing an Option Part 2

2008-10-08T00:00:00+00:00

In my previous post I discussed creating an Option style construct for C#/.Net. This post is a followup with the complete code snippet. It’s been updated in response to several bits of feedback I received. Namely

Option is now a struct vs. class
Added equality metrics
Allowing implicit conversion T ‘> Option

As usual, this is available in the latest version of RantPack: http://code.msdn.microsoft.com/RantPack

[Immutable]
[Serializable]
[SuppressMessage("Microsoft.Naming", "CA1716")]
public struct Option<T> : IEquatable<Option<T>>
{
    private readonly T m_value;
    private readonly bool m_hasValue;

    public static Option<T> Empty
    {
        get { return new Option<T>(); }
    }

    public bool HasValue
    {
        get { return m_hasValue; }
    }

    [DebuggerDisplay("m_value")]
    public T Value
    {
        get
        {
            if (!HasValue)
            {
                throw new InvalidOperationException("Option does not have a value");
            }

            return m_value;
        }
    }

    public T ValueOrDefault
    {
        get { return m_hasValue ? m_value : default(T); }
    }

    public Option(T value)
    {
        m_hasValue = true;
        m_value = value;
    }

    #region Operators

    [SuppressMessage("Microsoft.Usage", "CA1801")]
    [SuppressMessage("Microsoft.Usage", "CA2225")]
    public static implicit operator Option<T>(Option option)
    {
        return Option<T>.Empty;
    }

    [SuppressMessage("Microsoft.Usage", "CA1801")]
    public static implicit operator Option<T>(T value)
    {
        return new Option<T>(value);
    }

    public static bool operator ==(Option<T> left, Option<T> right)
    {
        return left.Equals(right);
    }
    public static bool operator !=(Option<T> left, Option<T> right)
    {
        return !left.Equals(right);
    }

    #endregion

    #region IEquatable<T> Members

    public bool Equals(Option<T> other)
    {
        if (other.HasValue != this.HasValue)
        {
            return false;
        }

        // Both don't have a value
        if (!other.HasValue)
        {
            return true;
        }

        return EqualityComparer<T>.Default.Equals(m_value, other.Value);
    }

    #endregion

    #region Overrides

    public override bool Equals(object obj)
    {
        if (obj is Option<T>)
        {
            return Equals((Option<T>)obj);
        }

        return false;
    }

    public override int GetHashCode()
    {
        if (!HasValue)
        {
            return 0;
        }

        return EqualityComparer<T>.Default.GetHashCode(m_value);
    }

    #endregion
}

[Immutable]
[Serializable]
[SuppressMessage("Microsoft.Naming", "CA1716")]
public sealed class Option
{
    private static Option s_empty = new Option();

    private Option()
    {
    }

    public static Option<T> Create<T>(T value)
    {
        return new Option<T>(value);
    }

    public static Option Empty
    {
        get { return s_empty; }
    }
}

Unfold

2008-10-07T00:00:00+00:00

F# has a handy method called Unfold. Think of it as the logical opposite of an Aggregate function. Aggregates take a sequence of elements and convert them to a single element. An unfold method will take a single element and turn it into a (potentially infinite) sequence of elements.

The API is straight forward. It takes two parameters

A seed/start value
A function which takes in a seed value and returns either an Empty option to indicate the end of a sequence or a value of Type tuple with two arguments. The first is the next element in the sequence and the second is the next seed value passed into the function.

This is a very powerful function which allows you to quickly build all sorts of interesting functions. Lets look at a potential implementation of Enumerable.Range as an unfold expression.

public static IEnumerable<int> Range(int start, int count)
{
    return Enumerable.Unfold(
        start,
        x => x - start < count
            ? Option.Create(Tuple.Create(x, x + 1))
            : Option.Empty);
}

Not as efficient as the framework version of Range but a good mental exercise to get your head around using Unfold. Implementing this method in C# is fairly straight forward.

public static IEnumerable<TResult> Unfold<TSource, TResult>(
    TSource state, 
    Func<TSource, Option<Tuple<TResult, TSource>>> func)
{
    if (func == null)
    {
        throw new ArgumentNullException("func");
    }

    return UnfoldHelper(state, func);
}

private static IEnumerable<TResult> UnfoldHelper<TSource, TResult>(
    TSource state, 
    Func<TSource, Option<Tuple<TResult, TSource>>> func)
{
    do
    {
        var result = func(state);
        if (!result.HasValue)
        {
            break;
        }

        yield return result.Value.First;
        state = result.Value.Second;
    } while (true);
}

Functional C#: Providing an Option

2008-10-06T00:00:00+00:00

Sorry for the terrible pun in the title. I wanted to blog about developing an F# style Option class for C# and I couldn’t resist.

The basics of an Option class are very straight forward. It’s a class that either has a value or doesn’t. It’s almost like nullable but for every type and allows for nulls to be a valid value. Here’s a straight forward Option class I coded up.

public sealed class Option<T> {
    private readonly T m_value;
    private readonly bool m_hasValue;
    public static Option<T> Empty {
        get { return new Option<T>(); }
    }
    public bool HasValue {
        get { return m_hasValue; }
    }
    public T Value {
        get {
            if (!HasValue) {
                throw new InvalidOperationException("Option does not have a value");
            }
            return m_value;
        }
    }
    public Option(T value) {
        m_hasValue = true;
        m_value = value;
    }
    private Option() {
        m_hasValue = false;
        m_value = default(T);
    }
}

public sealed class Option {
    public static Option<T> Create<T>(T value) {
        return new Option<T>(value);
    }
}

I modified a bit of terminology to be more consistent with other frameworks I use (Some/None -> Value,HasValue). It’s succinct, generic and has type inference friendly create functions. Or does it?

Lets consider a function which has a return type of Option. Case 1 is Option with a value. There is a type inference friendly Option.Create method which makes for a simple return expression. No types needed.

Option<int> SomeMethod() {
    return Option.Create(42);
}

Now lets consider Case #2, None. Here there is no handy inference method because what would we use for inference. There is no variable with a Type to use so we are forced to be explicit about the type.

Option<int> SomeMethod2() {
    return Option<int>.Empty;
}

In this case we’re not so bad off because we’re dealing with a simple type. But what about more complex types’ Consider for example a hypothetical Unfold method. The termination expression would be Option<Tuple<int,string».Empty. Anonymous types are even worse since they are unnamable and can not ever be the source of an option with this design. This really pales in the usage category when compared with F#.

Lets see if we can do better.

First we could consider a design where we have a static creation method Empty which takes variables that aren’t ever used. This will give us the benefit of type inference but at the expense of an API which is faulty to the core. It forces the user to create parameters that aren’t ever used. Definitely not a good design.

This leaves us with using a solution that doesn’t involve variables of the necessary type. This essentially forces us into a non-generic solution since we need variables for type inference. This non-generic Option won’t be compatible with our generic return type. But wait, what will the compiler do if two expressions have conflicting types’ Eventually it will attempt to perform a conversion. So if we make our non-generic empty Option convertible to any generic empty Option we can use the compilers type safety to our advantage.

Definition a non-generic empty Option is straight forward.

public sealed class Option {
    private static Option s_empty = new Option();
    private Option() {
    }
    public static Option Empty {
        get { return s_empty; }
    }
    public static Option<T> Create<T>(T value) {
        return new Option<T>(value);
    }
}

Using a private constructor allows us a high degree of confidence that any Option instance hanging around came from our Empty property and hence represents an empty option. Now all we need to do is define a conversion on Option. Essentially we want to say that any non-generic Option is convertible to this instance. Add the following to Option

public static implicit operator Option<T>(Option option) {
    return Option<T>.Empty;
}

Now, we can use an empty option in any generic scenario without having to specify ugly type parameters.

Option<int> SomeMethod3() {
    return Option.Empty;
}

When Is A Long Not A Long

2008-10-02T00:00:00+00:00

Answer: When PInvoke is involved.

I ran across a common error today on stackoverflow regarding P/Invoke that is worth blogging about. The question regarded the translation of a native API with a parameter of type LONG. The user mistakenly used the .Net long type as the parameter. The error is that a C++ LONG is not the same as a .Net long.

When talking about types and PInvoke it’s easier to discuss byte size and signed-ness than type names. Otherwise confusion around long and short crop up. Really there are four integer byte sizes each of which can be signed or unsigned: 1,2,4 and 8

The problem the user encountered is the C++ long is 4 byte signed and .Net long is 8 byte signed. PInvoke requires the parameters to have the same size. Below is a quick table of the various types in C++ and .Net.

1 byte
- C++ - char, __int8, BYTE, BOOLEAN
- .Net - byte
2 byte
- C++ - wchar, __int16, short, WORD
- .Net - char, short
4 byte
- C++ - int, LONG, long, __int32, DWORD
- .Net - int
8 byte
- C++ - __int64, LONGLONG, DWORDLONG, LARGE_INTEGER
- .Net - long

Based on this table when translating a C++ LONG, you should use a .Net int.

Edit1: Moved C++ short to 2 byte, added several other C++ types. (thanks Raymond)

Learching Definition To Live Search

2008-10-01T00:00:00+00:00

Unfortunately when creating Live Search, the live team did not use a term which could easily be converted to a verb (i.e. google and google-ing). The term “live searching” doesn’t really flow well in a hallway conversation nearly as well as google-ing.

I live searched that movie John was talking about …

I considered for awhile just using the “googling” term for both using live search and google. But this wouldn’t allow me to quickly correlate the goods and bads of each search engine. I needed a new term.

Thus learching was born. It’s short for “Live sEARCHing”. When pronouncing it should rhyme with “searching.”

Anyone else with a better term?

Pinvoke Interop Assistant On Channel 9

2008-09-11T00:00:00+00:00

Beth Massi dropped by my office a couple of months ago and we did a channel 9 video on the PInvoke Interop Assistant. Mainly an overview of the product and a bit of a tutorial. Check out the video here.

http://channel9.msdn.com/posts/funkyonex/The-P-Invoke-Interop-Assistant/

Euler 2

2008-09-11T00:00:00+00:00

Problem #2 came quite a bit faster. The yield syntax in F# is very similar to the C# iterator syntax and made translating this sample a breeze.

As a commenter posted, in problem #1, I would’ve been better served to use Seq.filter as opposed to Seq.choose for the purpose of filtering list. I’ve applied that here. One downfall of learning a new language is finding the right API. I spent a bit of time reading through the Seq documentation looking for the equivalent of the Where extension method and could only find chose. Lesson learned

// Find the sum of all the even-valued terms in the sequence which do not exceed four million.
let problem2() =
let rec fib prev cur = seq {
yield prev
if cur < 4000000 then
yield! fib cur (prev+cur)
}
fib 1 2
|> Seq.filter (fun x -> 0 = x % 2)
|> Seq.sum

Debugging F# lists

2008-09-10T00:00:00+00:00

One of the lacking’s of the latest F# CTP is debugger visualization support for the built-in list types. Viewing a list in the debugger is decidedly tedious compared to the mscorlib collection classes. Take the following quick code sample

module Main =
    do
        let l1 = [0..4]
        let l2 = List.map (fun a -> a.ToString()) l1
        let l3 = new System.Collections.Generic.List<int>()
        List.iter (fun i -> l3.Add(i)) l1
        MainModuleTemp.Main()   // Breakpoint here

Hit F5 in a F# console application and you’ll get the following display.

Notice how the elements of the mscorlib List<> are immediately visible. Getting to the data in a F# list is possible but it takes a lot more clicks than the mscorlib version. This doesn’t appear to be an oversight on the F# team either. The expansion of mscorlib List is controlled by the DebuggerTypeProxy attribute on the class definition. If you fire up Reflector and dig into Fsharp.Core.dll and navigate to List you'll notice it indeed has a DebuggerTypeProxy entry which is well formed and points to ListDebugView.

ListDebugView is essentially identical to the one for mscorlib List. So what gives' The bug appears to be in the accessibility of the constructor. Even though it's not explicit in the documentation of DebuggerTypeProxy, it appears that the target type must have a single argument constructor which is public. The one for ListDebugView is internal.

Normally this would be an easy enough problem to work around. Add an assembly level attribute of type DebuggerTypeProxy pointing to List and a modified version of ListDebugView. Unfortunately that will not work in this case. The debugger will prefer DebuggerTypeProxy instances added directly to a type over ones defined at an assembly level.

That is, except for two cases. The debugger will give precedence to assembly level attributes which are defined in an assembly named autoexp.dll and placed in one of the following two locations

Visualizers folder for the current user. One my machine it is C:\Users\jaredp\Documents\Visual Studio 2008\Visualizers
Devenv global visualizer folder. C:\Program Files\Microsoft Visual Studio 9.0\Common7\Packages\Debugger\Visualizers\Original

If you navigate to either of these directories you will find both the default autoexp.dll and the source code used to compile it. It’s got quite a few entries you may want to add in a modified version. Adding a new ListDebugView here is possible but lets do it in F# instead.

Since autoexp.dll has predecence all we need to do is build a new version which has the appropriate debugger attributes for the F# collections. Fire up a new class library project named autoexp and have it output to either of the directories listed above. Below is a sample definition to get you started.

#light
open System.Diagnostics

module Main =
    type ListProxy<'a>(l:List<'a>) =
        [<DebuggerBrowsableAttribute(DebuggerBrowsableState.RootHidden)>]
        member this.Items = 
            Array.of_list l
            
    [<assembly: DebuggerDisplayAttribute("{Length}", Target=typeof<List<int>>)>]
    [<assembly: DebuggerTypeProxyAttribute(typeof<ListProxy<int>>, Target=typeof<List<int>>)>]
    do 
        ()

Don’t be alarmed at the typeof<List>. The visualizer will work for any generic binding of List. In fact, reflector confirms that this attribute will be emitted with the type pointing at the unbound List instead of List. My lack of F# skills is failing me as to why. I'd love to cry bug but I've found crying bug at a compiler is usually ... wrong.

In either case, once you build this and place in the appropriate folder, you should find the visualizations for List<> much more accessible.

When To Use Type Inference

2008-09-09T00:00:00+00:00

Occasionally the debate will come as to when it’s OK to use type inference in order to declare a variable. There appear to be three groups in this debate.

Whenever it’s possible
Only when it’s absolutely clear what the type is
Never, type inference is evil

I fall into camp #1 and here are my reasons

It does not reduce type safety. This doesn’t allow for any late binding, type unsafe functions or the like. It simply lets the compiler chose the type for you.
It will actually increase type safety in your code. The best example of this is the foreach statement on non-generic IEnumerable instances. These foreach statements are all technically unsafe because the compiler must do a cast of the Current member under the hood. This declaration looks no different than the type safe generic version of IEnumerable. Using var will force you to write an explicit cast.’?

foreach (SomeType cur in col)

foreach ( var cur in col.Cast())
Maintains the principles of DRY. This is mostly true for cases where you have an explicit constructor on the RHS.
For some types, this is a requirement in order to use the type (anonymous types for instance). I’m a big fan of consistency and since I must have some instances use type inference, I’d like to use them everywhere.
Makes refactoring easier. I re-factor, a lot. I constantly split up or rename types. Often in such a way that refactoring tools don’t fixup all of the problems. With var declarations I don’t have to worry because they just properly infer their new type and happily chug along. For explicit type cases I have to manually update all of the type names.
Less typing with no loss of functionality.

The best argument I’ve heard against type inference is that it reduces readability since you can’t look at a variable and know it’s type. True, but just hover over the declaration and the IDE will display the type. Yes this is not possible with a non-IDE editor but how often do you use one?

Euler And F

2008-09-08T00:00:00+00:00

I’ve been looking for some new problems to work on in F# to get more comfortable with the language. I’ve been rather slack of late because of other projects but I had a little bit of time this week. I decided it would be fun to join the crowd and play away at the problems on the project euler site. That being said, answer #1.

module Euler =
    let problem1() =
        let test i = 
            match (0 = i % 3) || (0 = i % 5) with
                | true -> Some i
                | false -> None
        let targetSeq = Seq.choose test [0..999]
        Seq.sum targetSeq

Where Does The Go

2008-09-04T00:00:00+00:00

This is a more amusing than functional debate I enter into from time to time. On a line where you declare a pointer type in C++, where should the * go?

Next to the type (i.e. Type* p1;)
Next to the variable name (i.e. Type *p1;)
Who cares

For the moment lets ignore #3 (after all they don’t care). I’m a firm believer in #1. After all * is a part of the type of the variable, not the name and therefore should be closer to the type.

#2 believers disagree with this notion. They believe the * is a part of the individual variable’s type and not the actual type. This is technically correct and can be demonstrated with the following code

Type* p1, p2;

The type of p2 is of course Type and not Type*. Therefore they argue, #2 is the superior way

This is true but I’m also a firm believer in don’t declare multiple variables in a single declaration statement while coding in C++ unless the type has a user defined constructor. Namely to avoid situations just like this.

F Ctp First Impressions

2008-09-03T00:00:00+00:00

I had a little bit of time this weekend to download the F# CTP. The IDE portion of the CTP is a huge improvement over the previous CTP. In particular the intellisense engine feels much more smooth, is available in new places and has much better results. Project engine also has better integration. Debugging support is much better.

Overall the IDE now really feels like a … well IDE. I felt like the previous CTP was forced into Visual Studio and had much more in common with the old C++ IDE than VB and C#. This time around the IDE is just more smooth and more closely resembles other managed integrations. Definately worth the upgrade.

A Lesson In Serialization

2008-09-02T00:00:00+00:00

A few days ago, I recklessly added a [Serialization] attribute to a few of my immutable collection types. I needed to pass data between AppDomain’s and adding [Serialization] was the quick and dirty fix. Compiled, ran and I didn’t think much about it.

Luckily I was updating some unit tests last night and I remembered this and added a couple of serialization sanity tests. Most of the tests passed first time but for my ImmutableStack class¹ was throwing an exception. Well, it was actually my ImmutableQueue but it was failing in one of the inner ImmutableStack instances. The test code was fairly straight forward

var stack = ImmutableStack.Create(new int[] { 1, 2, 3 });
using (var stream = new MemoryStream()) {
    var f = new BinaryFormatter();
    f.Serialize(stream,stack);
    stream.Position = 0;
    var obj = f.Deserialize(stream);
    var stack2 = (ImmutableStack<int>)obj;
    var stack3 = stack2.Reverse();
}

I did a bit of digging and discovered the exception was coming from the stack2.Reverse() call. Jumped through the code and didn’t see much wrong. I had several existing tests around ImmutableStack.Reverse() and I couldn’t see why Serialization would make any difference.

public ImmutableStack<T> Reverse() {
    var r = ImmutableStack<T>.Empty;
    var current = this;
    while (current != ImmutableStack<T>.Empty) {
        r = r.Push(current.Peek());
        current = current.Pop();
    }

    return r;
}

Can you see what’s wrong with the code?

It took me a few minutes of debugging and frustration. The bug is in the while loop conditional. Until you introduce serialization this code is just fine. ImmutableStack.Empty is a static readonly declaration. The code implementation only allows for one to be created so it a singleton and equality can be done with a quick reference check.

private static readonly EmptyImmutableStack s_empty = new EmptyImmutableStack();

public static ImmutableStack<T> Empty {
    get { return s_empty; }
}

Unfortunately serialization breaks the assumption that EmptyImmutableStack is a singleton. The EmptyImmutableStack class is a singleton by convention only. It’s a private nested class that’s only instantiated once per AppDomain. There is nothing stopping the CLR or Serialization for that matter from creating a second instance. In the case of deserialization that’s exactly what happens. The serializer isn’t built to recognize this pattern and instead simply creates a new instance of EmptyImmutableStack upon deserialization.

This essentially prevents you from safely using a functional style Empty pattern inside a serializable collection.

The fix is simple enough, alter the conditional to be (!current.IsEmpty).

The version of ImmutableStack I’m using is heavily based off of Eric Lippert’s implementation. ↩

F Ctp Released

2008-08-29T00:00:00+00:00

A new version of the F# CTP was released today. There are a lot of new improvements to the language and project system. Check it out here

Disabling Jit Optimizations While Debugging

2008-08-29T00:00:00+00:00

If you’ve ever been debugging a managed app, only to be unable to evaluate any of the locals or parameters because the code was “optimized”, check out the article below. It shows a quick trick to disable optimizations by way of a .ini file. This is great because it doesn’t force you to recompile the application and takes only seconds to implement.

The short version is create an .ini file (i.e. myapp.ini) with the following contents.

[.NET Framework Debugging Control]
GenerateTrackingInfo=1
AllowOptimize=0

This has really saved me time debugging recently. It’s been blogged about by several others but given that I’ve had to search for this solution 3 times in as many weeks, I figured blogging about it would make it easier to find next time :)

http://msdn.microsoft.com/en-us/library/9dd8z24x.aspx

5 Things Hollywood Thinks Computers Can Do

2008-08-27T00:00:00+00:00

Reading another Jtenos’ blog, I came across this article about things Hollywood thinks can be done with computers. Humorous read.

http://www.cracked.com/article_15229_5-things-hollywood-thinks-computers-can-do.html

Comments

2008-08-26T00:00:00+00:00

If only people spent as much time writing comments as they did speaking to the evils of comments. Everything from useless, inaccurate, to many comments make code unreadable, you should code better … etc.

I haven’t ever looked at a piece of code and thought “wow, way too many comments” or “that code would be beautiful if it weren’t for the comments.”

Do the next guy a favor, comment your code.

Simple Class For Tests Involving A Synchronizationcontext

2008-08-25T00:00:00+00:00

Recently I had to test a class which heavily depended upon a SynchronizationContext. This threw me off for about half an hour as I didn’t want to write multi-threaded unit tests. Multi-threaded code is difficult enough without adding needless threads.

The solution I came up with is simple and gives the unit test a large degree of control over the execution of posted delegates. The resulting tests were much easier to code and understand.

public sealed class TestSynchronizationContext : SynchronizationContext {
    private List<Tuple<SendOrPostCallback, object>> m_pending 
        = new List<Tuple<SendOrPostCallback, object>>();

    public override void Send(SendOrPostCallback d, object state) {
        d(state);
    }

    public override void Post(SendOrPostCallback d, object state) {
        m_pending.Add(Tuple.Create(d, state));
    }

    public void RunAllPosted() {
        m_pending.ForEach(x => x.First(x.Second));
    }
}

Clr Changes In 3 5 Sp1

2008-08-22T00:00:00+00:00

If you haven’t come across this post yet about inlining changes in 3.5 SP1 it’s a good read. The short version is, value types will have better performance.

http://blogs.msdn.com/clrcodegeneration/archive/2007/11/02/how-are-value-types-implemented-in-the-32-bit-clr-what-has-been-done-to-improve-their-performance.aspx

Don T Speak Geek To Your Family

2008-08-20T00:00:00+00:00

In planning my recent vacation an email thread was started about who would be picking up the groceries. Most importantly, who would be picking up the drinks. My brother was quick to volunteer. I said in no uncertain terms I didn’t want any of drinkA [^1]. In the past my brother was known to buy large quantities of drinkA.

Or at least I thought I said it in no uncertain terms. As we arrived I opened the fridge to find a large quantity of drinkA. I stared in disbelief and confronted my brother. He responded with “yeah, I thought it was weird that your email said to buy that but since you clearly wanted it …”?? After several other family members said the same, I placed the blame on a busy week at work and a simple typo.

Luckily my wife brought up the email in question as proof of my mistake. The line read

Jason we want beer. For the purpose of the beach beer != drinkA

To most people reading this entry the meaning is as clear as it can be. Don’t buy drinkA. Unfortunately the rest of my family are not programmers and read this as an emphatic equals sign.

Lesson learned.

[^1] The particular drink in question is not important.

Immutablestack In F Part 2

2008-08-19T00:00:00+00:00

I’m a bit busier than I thought I would be after returning from vacation. But I had a little bit of time to play around with the implementation again today. Thanks to all the suggestions in the comments from the previous post.

This version has a couple of improvements including

Removing the ambiguous constructor
More efficient All() implementation

Remaining issues:

Need to make it generic :)
Still exposing a type union. Great for F# but it produces somewhat awkward looking metadata for non-F# languages to consume

#light

namespace Col  
   type ImmutableStack =  
       | EmptyStack  
       | Value of int * ImmutableStack  
       static member Empty = EmptyStack  
       static member Create( l:#seq<int> ) =  
         let s = ref ImmutableStack.Empty  
         l |> Seq.iter (fun n -> s := (!s).Push(n) )  
         !s  
       member x.Count =  
         let rec count t (cur:ImmutableStack) =  
           match cur.IsEmpty with  
               | true -> t  
               | false -> count (t+1) (cur.Pop())  
         count 0 x  
       member x.IsEmpty =  
         match x with  
           | EmptyStack -> true  
           | _ -> false  
       member x.Push(y) = Value(y,x)  
       member x.Peek() =  
         match x with  
           | EmptyStack -> failwith "ImmutableStack is empty"  
           | Value (v,_) -> v  
       member x.Pop() =  
         match x with  
           | EmptyStack -> failwith "ImmutableStack is empty"  
           | Value (_,n) -> n  
       member x.Reverse() =  
         let rec doBuild (cur:ImmutableStack) (building:ImmutableStack) =  
           match cur.IsEmpty with  
               | true -> building  
               | false -> doBuild (cur.Pop()) (building.Push(cur.Peek()))  
         doBuild x ImmutableStack.Empty  
       member x.All() =  
         match x with  
           | EmptyStack-> Seq.empty<int>  
           | Value (v,n) -> seq {  
               yield v  
               yield! n.All() }

First Msdn Article

2008-08-18T00:00:00+00:00

The August issue of MSDN magazine will be carrying an article I wrote this spring. In it I toy around with using the deferred execution and lazy evaluation properties of LINQ to create more responsive UI code.

You can view the article here.

As usual I appreciate any feedback you have.

Immutablestack In F

2008-08-15T00:00:00+00:00

When learning a new language I find it very instructive to re-code certain structures from my well used libraries into the new language. It gives a great basis for comparison in terms of ease of implementation, expressiveness of the language and sheer ease of implementation. So on that note I set out today to build an ImmutableStack implementation in F#. This is based off of my C# implementation in RantPack.

Below is the initial implementation. This is my first non “hello world” data structure in F#. I found it surprisingly easy to implement and I’m really enjoying the language. The biggest stumbling block was getting the type union correct and dealing with my compulsion to use “null” for end of stack instead of a value.

After playing around with it a bit I’m left with the following questions/hangups. Most of these will just fall into the category of “I’m starting out with a new language so I’m still hung up on the syntax in places.”

I consider Node to be an implementation detail and ideally would like to make it a private nested class if possible
The constructor still allows for invalid data combinations (but will throw) 1. Ex: ImmutableStack None ImmutableStack.Empty()
Can I get ImmutableStack.Empty to be a property instead of a function?
In All(), that can’t be the most efficient way to build up a sequence.

#light

type Node = 
    | Empty
    | Value of int * ImmutableStack

and ImmutableStack(?v:int, ?n:ImmutableStack) = 
    let data = match (v,n) with
                | (Some v, Some n ) -> Value (v,n)
                | (Some v, None) -> Value (v,ImmutableStack.Empty())
                | (None, None) -> Empty
                | _ -> failwith "invalid combination"
    static member Empty() = ImmutableStack()
    member x.IsEmpty() = 
        match data with
            | Empty -> true
            | _ -> false
    member x.Push(y) =
        match data with 
            | Empty -> ImmutableStack(y, x)
            | Value _ -> ImmutableStack(y, x)
    member x.Peek() =
        match data with
            | Empty -> failwith "ImmutableStack is empty"
            | Value (v,_) -> v
    member x.Pop() =
        match data with 
            | Empty -> failwith "ImmutableStack is empty"
            | Value (_,n) -> n
    member x.All() =
        match data with 
            | Empty -> Seq.empty
            | Value (v,n) -> Seq.append (Seq.singleton v) (n.All())
            
let rec printStack (s:ImmutableStack) =
    match s.IsEmpty() with
        | true -> printfn "Empty"
        | false -> 
            printfn "%d" (s.Peek())
            printStack (s.Pop())

let s1 = ImmutableStack.Empty()
let s2 = s1.Push(42).Push(56).Push(62)
let s3 = ImmutableStack 42
let s4 = s3.Pop()

printStack s1
printStack s2
printStack s3
printStack s4

Learning A New Language F

2008-08-14T00:00:00+00:00

Time again to learn a new language. I enjoy picking up a new language every now and again. It’s a great way to branch out your skill set and usually leads to new programming techniques in languages that are a part of professional or hobby life.

But why bother’ The popular languages have a lot of common elements : functions, types, etc … With all of the common elements what can learning a new language really give you?

It’s true that there are a lot of common elements (especially for .Net based languages). But what really interests me are the features which make the individual language extremely productive. Most languages have a feature or two that really makes it worth learning. They greatly increase the productivity of the developer. Often full libraries are designed around these features. For instance

C++ and?? Templates
- This is somewhat special in that it’s extraordinarily powerful for both good and evil
VB and late binding
C# and iterators

What’s great about learning these features is I can often import them into my current heavily used languages. For instance, even though C#/VB don’t directly support functional style programming, I’ve spent a bit of time over the last few months building libraries that make it fairly easy to incorporate them into either language. This added a deal of flexibility into my programs and a real functional feel.

Lately I’ve been focusing on immutable/persistant data structures, lambda expressions and tuples. This makes F# a natural choice for a language to learn. While I tend to program in a somewhat functional style this will be the first functional language I’ve heavily invested my time in. For awhile I did a bit of Haskell but nothing serious.

I intend to blog about the joys and not so joyful moments of this process.

Vacation

2008-07-29T00:00:00+00:00

I’m embarking upon another cross country road trip this week followed by a much needed vacation. I will be actively blogging during this time (lots of backlogged material to get through) but without a decent internet connection the live blog will be … quiet.

Rantpack Update 2 0 0 2

2008-07-23T00:00:00+00:00

Version 2.0.0.2 released.

Summary: RantPack is a utility library I maintain and actively use. The main themes of this library are functional programming, patterns, immutable/pressitent collections, future and other threading primitives. I’ve placed the code and binaries on http://code.msdn.microsoft.com/RantPack for educational and sharing purposes.

Release Link: https://code.msdn.microsoft.com/Release/ProjectReleases.aspx?ProjectName=RantPack&ReleaseId=1306

Release Notes:

Renamed Tuple properties to be more consistent
- Item0, Item1 instead of A,B
Added a Match operator
Made Enumerable more compatible
Bug fixes

Live Mesh Accepting More Users

2008-07-17T00:00:00+00:00

If you haven’t tried out live mesh yet I really encourage you to try. It’s an amazing well designed product and incredibly useful.

It’s still classified as a tech preview but now they are taking on additional users (doubling the current user base in size).

http://blogs.msdn.com/livemesh/archive/2008/07/16/live-mesh-still-in-techpreview-but-ready-for-more-users.aspx

Don T Mix Using Statements And Lambda Expressions

2008-07-16T00:00:00+00:00

Title pretty much says it all but what good is a rule without any explanation. The main issue here is that at the core, using statements and lambda expressions both alter variable lifetimes. Unfortunately they alter the lifetime in different directions. Using will shorten the life time of a variable to the specified block. This is a somewhat artificial way because the object is still technically alive but can’t be trusted to do anything. Lambda expressions take a variable limited to a specific scope and extends their lifetime to potentially be that of a heap value. Anytime two features alter the attribute of a variable in different directions, they can probably cause problems when used in conjunction.

Take the following contrived but real example.

static Future<int> Example() {
    using (var obj = new MyDisosableObject()) {
        return Future.Create(() => obj.SomeFunction());
    }
}

Yet Another Rule For Equality

2008-07-11T00:00:00+00:00

“If you implement equality in a child class, including operators, you must implement the equality operators in the base class.”

Unfortunately this is another case of learn the hard way but makes sense when you think about it. The below code snippet is an example of the problem that I hit. Even though I have equality properly defined in Child, the equality check goes through Parent. As such the C# compiler will perform the default comparison which is reference equality.

The simple fix is to add the operator ==/!= definitions to Parent which call through EqualityComparer.Default. This will end up calling obj.Equals and equality will function correctly.

While this is intuitive when you think about it, it’s an easy trap to fall into. It would be nice if there was a Compiler/FXCop warning here.

class Parent {
}
class Child : Parent{
    public readonly int Field1;
    public Child(int value) {
        Field1 = value;
    }

    public override int GetHashCode() {
        return Field1;
    }
    public override bool Equals(object obj) {
        var other = obj as Child;
        if (other == null) {
            return false;
        }
        return other.Field1 == Field1;
    }
    public static bool operator ==(Child left, Child right) {
        return EqualityComparer<Child>.Default.Equals(left, right);
    }

    public static bool operator !=(Child left, Child right) {
        return !EqualityComparer<Child>.Default.Equals(left, right);
    }
}

class Program {
    static void Main(string[] args) {
        Child child1 = new Child(42);
        Child child2 = new Child(42);
        Parent parent1 = child1;
        Parent parent2 = child2;
        bool isChildEqual = child2 == child1;       // True
        bool isParentEqual = parent1 == parent2;    // False
    }
}

Book Review The Pragmatic Programmer

2008-06-25T00:00:00+00:00

Finished reading The Pragmatic Programmer a few weeks ago. I’ve been slacking off on the book review but better late than never.

This is a good book for people desiring to be a more pragmatic programmer. At ~270 pages it’s a quick read and loaded with content.’? This book has several predominant themes

Personal responsibility in programming
Pragmatic approaches to problems
Paranoid programming

My personal favorite has to be #3. If you don’t trust anything, including yourself, you’re going to have fewer and fewer unexpected bugs in your code.

Along those lines is the best chapter in the book: Pragmatic Paranoia. This chapter details the virtues of design by contract, aggressive use of asserts and the virtues of crashing the instant a problem is discovered.

The next best chapter in the book, “A pragmatic approach”, deals with the general approach to programming. You’ve likely heard these all in pieces before but the authors do a great job of organizing and presenting the material.

One particular approach new to me is Tracer Bullets. This is similar to prototyping but with a very different aim. Prototypes should be used to quickly test out a subset of a bigger solution while ultimately being throw away code. Trace Bullets are also designed to give a quick solution but also to be the final product. They focus on getting the basic framework implemented and simple implementations for pieces in order to give the user a working product. Feedback will create new iterations of the product on the same framework.

While overall a good book there are some parts that advanced programmers will likely skip over. They deal with the usage of basic tools of a programmer: debugger, text editor, source code control. All great suggestion/requirements and reading for newer programmers though.

All in all a good book.

Citation:

Hunt, Andrew and Thomas, David. The Pragmatic Programmer. Addison Wesley Longmam, Inc, 2000

To Name A Tuple Value A Or Item0

2008-06-23T00:00:00+00:00

One of the parts of a Tuple implementation for RantPack I struggled the most with was naming. I tend to struggle with names quite a bit and there’s really no reason for it. It’s a combination of pickiness and … well there’s really no good reason. Pieces which can bother me range from

Don’t like the way it types. If I can’t type something easily it drives me nuts and I secretly desire to change it to a word that is easier to type. I have sadly spent a large amount of time on www.thesaurus.com looking for better words to type.
Looks funny. No real solid criteria here, just what I don’t like. Words sometimes look good the first time you type them but over time lose their appeal
Doesn’t match conventions. Ah, now we’re onto solid reasons. Names should match conventions.
Name is ambiguous. If the name doesn’t give a clear intention of the operation I do my best to change it.
FxCop violation. This bothers me less than others but if FxCop doesn’t like it I think a bit harder about naming it.

Now comes Tuples. Many implementations of tuples don’t even have names but instead rely on language constructs such as pattern matching to access the values. That’s not really the DotNet way of doing things though so I chose to give Tuple values names.

Originally I chose A for item 0, B for item 1, etc … This made sense at the time because it kept the syntax short. However I’ve grown to dislike this naming convention. Primarily because it violates #3 and #4 above. While A-E is perfectly clear for me I’ve talked with others who this confuses.

In addition, to keep the nameless syntax option I provided an indexer property. So values are available with a 0 based index. Since several languages expose this with the name “Item”, there are now really two names for a value. For example the first value is both tuple.A and tuple.Item[0].

After some thought I decided to name the values Item0->Item4 depending on the size of the tuple.

Making Code Display Better On Your Blog

2008-06-19T00:00:00+00:00

My blogging tool is Windows Live Writer and I use the “Insert From Visual Studio” plug-in to get pretty looking code into my postings. The generated code uses the <pre> tag for formatting the elements.

Unfortunately my blog provider doesn’t always render this properly and will clip text that is too long. Ideally I would like to either

Wrap code that overflows the page
Put up a localized scroll bar for the code snippet

Luckily the plug-in puts the outer most <pre> tag into it’s own CSS class: code. This makes the problem easy with a CSS override.

pre.code {  
   overflow : auto;  
}

Enums Vs Adapters

2008-06-18T00:00:00+00:00

I like Enums and use them frequently for options and behavior. To an extent I use Enum’s to control behavior. For example

enum Kind {
    Kind1,
    Kind2,
    Kind3
}

class Example {
    private Kind m_kind;

    public int SomeAction() {
        switch (m_kind1) {
            case Kind.Kind1:
                return ActionForKind1();
            case Kind.Kind2:
                return ActionForKind2();
            case Kind.Kind3:
                return ActionForKind3();
            default:
                throw new InvalidOperationException("Invalid Kind");
        }
    }
}

This is an acceptable pattern and use for enums. However if you take a step back, what I’ve actually done here is use an enum to implement an adapter pattern. I’ve just been a bit lazy about it and not actually coded up the classes.

To an extent though this violates the principle of single use as Example now performs N different behaviors based upon the enum. But lets face it, if ActionForKindN() is just a simple 2 line function then is it really worth it to create and maintain an adapter pattern’ A purest would likely say yes but I’m more pragmatic and don’t believe so.

Once the functions reach a certain level of complexity though an adapter pattern is much more suitable. Over time I find that many of my similar patterns evolve to level.

Yet I struggle to define the point at which an adapter is suitable. After several recent experiences I started adapting the following rules. If any of them is violated then I switch from an enum based behavior to adapter based behavior.

The Action* method contains state
There are more than 2 functions which change their behavior based on the enum value
All methods in the class change their behavior based on the enum

Anyone have a better set?

Pinvoke Interop Assistant Now On Codeplex

2008-06-17T00:00:00+00:00

I’m happy to announce that the PInvoke Interop Assistant tool is now available on CodePlex. This includes the binaries, source code and the dev authored unit tests.

It is hosted on the more general CLR Interop Tools page.

http://www.codeplex.com/clrinterop

I will be actively maintaining this tool in the future and (hopefully) adding more features. There were several features we cut just before release due to QA costing that I would like to add back (including VS integration, wrapper functions).

Please post any suggestions you have for the tool on the site and we will definitely consider them.

Bit By Void

2008-06-16T00:00:00+00:00

Recently I got bit by void* again because of another C++ quirk I didn’t think through. I had a class which wrapped a void* which could be one of many different structs. The structs were POD and didn’t have any shared functionality hence I didn’t bother creating an inheritance hierarchy. Unfortunately I defined the structs like so

class C1 {
  struct S1 {
    int field1;
    float field2;
  };
  struct S2 {
    char field1;
  };
  ~C1() {
    delete m_pData;
  }
  void* m_pData; // Can be S1,S2,etc ...
}

Unfortunately this appeared to work fine for quite some time. Then after a couple of days of bug fixes I ended up with a memory leak which I quickly tracked down to a leaked COM object. Although C1 was at fault I didn’t suspect any changes to this class because after all it was working fine for some time and all I did was add a new field to one of the structs. If the structs were being successfully free’d before a new field shouldn’t change anything.

The field I added was of type CComPtr which exposed a greater problem in my code. Even though I properly delete the pointer in C1::~C1() I wasn't running the destructor on the pointed at data and instead I was just freeing the memory. Until I added a field which had a non-trivial destructor this wasn't a problem (still a bug though).

Why did this happen’ By deleting a void* and expecting a destructor to run what I’m really doing is asking C++ to behave polymorphicly. C++ as a rule won’t behave this way unless it is specifically asked to with inheritance and virtual.’? In the case of void*, it just won’t. The fix is to actually implement an inheritance hierarchy which supports polymorphism.

It’s just another rule that I need to remember when coding C++.

Deleting void* is dangerous, period.

Unfortunately C++ has too many of these rules and not enough enforcement.

Using Powershell To Make Sure Your Xp Machine Is Defragmented

2008-06-13T00:00:00+00:00

Quick script you can run at login to ensure that your XP machine is being defragmented. I chose 1:00 AM every evening but you can quickly alter that in the script. I have this script run as part of my regular set of configuration scripts to ensure that my XP machines are in good shape.

$script:title = "Xp Regular Degrag"  
if ( 5 -ne [Environment]::OsVersion.Version.Major ) {  
    return;   
}  
$found = schtasks /query | ?{ $_ -match "^\w*$title" } | test-any  
if ( $found ) {  
    return  
}  
# Set up the defrag task  
$task = "{0} {1}" -f (join-path $env:WinDir
"System32\defrag.exe"),$env:SystemDrive  
schtasks /create /ru system /tn $title /sc daily /st "01:00:00" /tr $task

Is There Anything In That Pipeline

2008-06-12T00:00:00+00:00

One operation I frequently perform is use a powershell pipeline to filter out a large set of data. Typically I don’t care what is in the result but rather is there actually anything left in the pipeline. I can’t find a good powershell built-in to perform this task so I use the following filter.

function Test-Any() {  
    begin {  
        $any = $false  
    }  
    process {  
        $any = $true  
    }  
    end {  
        $any  
    }  
}

Now I can easily write

$any = Some-Command | ?{ Some-Condition } | test-any

Book Review Facts And Fallacies Of Software Engineering

2008-06-11T00:00:00+00:00

I finished reading Facts and Fallacies of Software Engineering a few weeks ago. This is an excellent book and I recommend it for anyone who’s been in the industry for a few years.

Typically I don’t enjoy books of this type because I feel they are too preachy, written by people who aren’t on the front lines and don’t understand the day to day problems a programmer will face or are just too long winded. Neither appears to be the case here.

Mr Glass takes a very interesting approach to this book. In addition to listing good resources for every fact/fallacy he also offers a small (albeit less detailed) section on why people disbelieve in this particular item.

Top 3 facts which I think have the most value in my job

Fact 36: Programmer-created, built-in debug code is an important supplement to testing tools. The majority of bugs filed against code I own are not because of an invalid behavior. It’s much more likely that QA will hit an ASSERT I added to the code base which fire before the behavior difference. What’s encouraging is QA will often note: If I ignore the assert everything seems to work alright. Without a dev assert these bugs would have gone unnoticed until a much bigger problem hit.
Fact 43: Maintenance is a solution not a problem. ** Except for the end of a cycle, I constantly do maintenance work to parts of the code base. If it’s broken, doesn’t meet standards, uses bad resource management, etc fix it!?? Don’t wait for the bug.
Fact 49: Errors tend to cluster. ** Good QA members know this and actively exploit it. Unfortunately it affects my job since I’m on the receiving end :)

Top 3 facts which I think more people need to take to heart

Fact 6: New tools and techniques cause an initial loss of productivity/quality. Anyone who reads my blog knows that I am a huge tool guy (see the PowerShell tag). Tools come with a cost though. As productive as PowerShell and my other bag of tools make me they all came with a price tag. One that has been paid over many times but the initial ramp up did cost time.
Fact 19: Modification of re-used code is particularly error prone. In my experience the costs of this are typically underestimated. Mainly because the behavior depended on is not well understood.
Fact 38: Rigorous inspections should not replace testing. I heard an engineer say it best on the history channel “One test is worth one thousand expert opinions.”?? Inspections mean nothing if they are not backed up by tests. Even if the inspectors get it right, without a regression test there is no way to guarantee it will stay that way for the future.

Top 3 facts which most surprised me

Fact 3: Adding people to a late project makes it later. I heard this long before I read the book but I wanted to call it out because I remember just how surprised I was the first time I heard it.
Fact 25: Missing requirements are the hardest requirement errors to fix. Falls into the category of “haven’t thought about that one but makes sense.”??
Fact 30: COBOL is a very bad language, but all the others (for business applications) are so much worse. This stems more than anything else from my complete lack of knowledge about COBOL. It’s an often scorned language yet if it was so bad no one used it then no one would complain.

Citation:

Glass, Robert L. Facts and Fallacies of Software Engineering. Pearson, Education Inc, 2003

Code Is Not Self Documenting

2008-06-09T00:00:00+00:00

Nothing revolutionary about that statement. Yet I keep reading the opposite on various comment threads and message boards so I thought it a good idea to explore it again.

Code is not self documenting.

The “code is self document” argument often comes up when commenting conventions, patterns and overall usage is discussed. People who are typically against writing more than the existing set of comments will throw out the argument “if we need a comment then the code should be written to be self documenting.”

To at extent I agree with this. Code should not be obfuscated and the usage should be clear. I personally strive to make my implementations as clear as possible and enforce that belief on anyone who asks me for a code review.

Yet while you can write code so that an individual algorithm or function is close to self documenting you cannot write it in such a way that it will explain it’s greater purpose in a program. Only comments can do that. Self describing code can only describe itself, not it’s purpose in the bigger picture.

Comments serve to both 1) explain the algorithm and 2) explain the greater purpose of the algorithm in the program.

Yet people still cling to the code is self documenting mantra. In my experience there are several reasons for this belief.’? The first is that people have only worked on projects small enough that for most purposes they can be kept entirely in their mind [*]. Until you work on a big enough program #2 is not even a factory because you intimately understand how every function fits into the big picture.

Another is that they have never worked on a project with people they weren’t very familiar with. People you don’t know well or have worked with before will likely have different ways and practices of approaching a problem which you have not encountered before. What is obvious to them won’t be obvious to you. The bridge between these approaches are comments.

I’ve seen DRY (don’t repeat yourself) brought up as well [**]. The code clearly says what it does so adding a comment is just repeating yourself. I find that to be patently untrue. If we’re even having the conversation then the code is clearly not self documenting. Also in the cases when an individual function is documenting you will still run into #2.

Commenting your code benefits both people who are reading your code and yourself. You will eventually come to a point where you’ve forgotten what a particular piece of code did or how it fit into your bigger program. Your comments will save you.

[*] The temptation to say "kept in memory" here was huge but I avoided it.

[**] In general I think that DRY is a great approach to programming but I feel it's being taken to far here

Making Equality Easier

2008-06-03T00:00:00+00:00

Recently I’ve done a bit of posting about the difficulties of properly implementing equality in VB (and DotNet in general). While most of the problems can be fixed with a standard snippet the one really hard to implement issue is GetHashCode(). The rules for GetHashCode() are both simple and seemingly contradictory

If two objects are equal (via Equals) their GetHashCode() should be equal
GetHashCode() shouldn’t ever change

These rules imply that GetHashCode() is related to Equality. At a fundamental level though GetHashCode has nothing to do with Equality. Instead it is linked to bucketing and ultimately any hashing sturcture such as Dictionary, Hashtable, etc …

Unfortunately it is impossible to separate these two from an API perspective because it is ingrained into the BCL. There is a way to separate this out at a functionality level and still satisfy all of the rules of the GetHashCode() and Equals()

Public Overrides Function GetHashCode() As Integer
    Return 1
End Function

This absolutely maintains both of the rules for GetHashCode(). This allows you to completely separate Equality and GetHashCode() in your implementation while not breaking any BCL rules or assumptions.

Of course this does come with a trade off. As said before GetHashCode() is primarily used as a bucketing mechanism. It will cause the performance of bucketing collections such as Dictionary or Hashtable to drop from close to O(1) to O(N). But once again this is not a bug but a conscious trade off.

Rantpack Update

2008-06-02T00:00:00+00:00

I released a new version of RantPack today. Mostly this is a bug fix release with a couple of minor new features.

https://code.msdn.microsoft.com/Release/ProjectReleases.aspx?ProjectName=RantPack&ReleaseId=1119

Features

Added a way to shim Immutable collections to non-immutable interfaces to increase the level of interoperability. The collections are still immutable and throw an exception whenever one of the mutable APIs are called. However if the collection is used in an immutable fashion, such as data binding, CollectionUtility.Create* can be used to quickly create a wrapper

    var q = ImmutableStack<int>.Empty;
    CollectionUtility.GetRangeCount(1, 10).ForEach(x => q = q.Push(x));
    ICollection<int> list = CollectionUtility.CreateICollection(q);
    
    var m = ImmutableAvlTree<int, string>.Empty;
    CollectionUtility.GetRangeCount(1, 10).ForEach(x => m = m.Add(x, x.ToString()));
    IDictionary<int, string> map = CollectionUtility.CreateIDictionary(m);

Added an immutable queue named ImmutableQueue
Added more overloads to Immutable*.Create to allow more interoperability with BCL data structures.

Bugs

MutableTuple.GetHashCode() violated the contract for Object.GetHashCode() by allowing updates to mutate the hash code

Inkpot Color Scheme

2008-06-02T00:00:00+00:00

I was reading Charlie’s post on VS Color schemes and thought I would add my favorite to the list. InkPot is a color setting for vim which I adapted to VS awhile back. Very easy on the eyes.

You can download it here

Do While 0 What

2008-05-21T00:00:00+00:00

A recent check in of mine raised a few eye brows during reviews. I checked in a few macros which ended with/contained a “do{}while(0)” and people were curious as to why.

In my experience there are two main uses for it.

Insert an empty statement with no runtime penalty
Group a set of statements into a single statement followable by a semi-colon 1. do { Func1(); Func2() } while(0)

Macros are mutating little constructs which jump back and forth depending on the compile time options. Many macros compile to different expressions based on the options and in some cases they compile to nothing. This is where #1 really comes in hand. Imagine you have the following code.

if(someCondition)
MY_MACRO();

Compiling MY_MACRO to nothing will cause at the least a compile time warning since you will have essentially if(someCondition);. Instead you can use the do{}while(0) trick.

#if CONST_1
#define MY_MACRO() Func1()
#else
#define MY_MACRO() do{}while(0)

Another us is for stylistic reasons. Some camps don’t believe a ; should be used as a empty statement and use do{}while(0) instead. I tend to agree. Mainly because one of my early C professors had the same belief. The habit stuck around.

Update: Changed incorrect use of word expression -> single statement

Outdated Comments Are Better Than No Comments

2008-05-20T00:00:00+00:00

While investigating our locking infrastructure a few days ago I ran across an odd comment. I was looking at a particular usage of a lock and the comment said that “Using lock type X because we must pump messages here.”?? Contrarily the lock type being used most definitely did not allow message pumping.

After a quick history search on the code base I was able to track down the developer responsible for the discrepancy. He merely forgot to update the comment when making the change. However he was able to explain the history behind the comment and the switching of the locking type.

I’ve heard people argue in the past that they didn’t comment code because comments get out of date and lead developers in the wrong direction. They might do it correctly but they didn’t trust the next guy. Then the comment would be wrong and useless for the next developer.

I think this is a example of why that mentality is wrong.

Even though it was currently wrong it was historically accurate
The commented add good insight into the choice of locking mechanism
It added a bit of detail into the historical architecture of the code base.
It was a heck of a lot better than staring at a comment-less field that appeared to exist for no reason.

I’m not arguing that comments which are flat out wrong at the time they are authored are a valuable resource. Yet not commenting because you don’t “trust” the next guy is equally wrong.

Switching On Types

2008-05-16T00:00:00+00:00

One action I find frustrating in C# is where a particular action needs to be taken based off of the type of a particular object. Ideally I would like to solve this with a switch statement but switch statements only support constant expressions in C# so no luck there. Previously I’ve had to resort to ugly looking code like the following.

Type t = sender.GetType();
if (t == typeof(Button)) {
    var realObj = (Button)sender;
    // Do Something
}
else if (t == typeof(CheckBox)) {
    var realObj = (CheckBox)sender;
    // Do something else
}
else {
    // Default action
}

Yes I realize this isn’t the ugliest code but it seems less elegant to me over a standard switch statement and I find the casting tedious. Especially since it requires you to write every type twice.

What I want to say is “Given this type, execute this block of code.” So I decided to run with that this afternoon. Lambdas will serve nicely for the block of code and using type inference will allow us to avoid writing every type out twice. What I ended up with was a class called TypeSwitch with 3 main methods (see bottom of post for full code).

Do - Entry point where switching begins
Case - Several overloads which take a generic type argument and an Action to run for the object
Default - Optional default action.

I wrote a quick winforms app to test out the solution. I dropped some random controls on a from and bound the MouseHover event for all of them to a single method. I can now use the following code to print out different messages based on the type.

TypeSwitch.Do(
    sender,
    TypeSwitch.Case<Button>(() => textBox1.Text = "Hit a Button"),
    TypeSwitch.Case<CheckBox>(x => textBox1.Text = "Checkbox is " + x.Checked),
    TypeSwitch.Default(() => textBox1.Text = "Not sure what is hovered over"));

Notice that for check box I was able to access CheckBox properties in a strongly typed fashion. The underlying case code will optionally pass the first argument to the lambda expression strongly typed to the value specified as the generic argument.

There are a couple of faults with this approach including Default being anywhere in the list but it was a fun experiment and works well.

TypeSwitch code.

static class TypeSwitch {
    public class CaseInfo {
        public bool IsDefault { get; set; }
        public Type Target { get; set; }
        public Action<object> Action { get; set; }
    }

    public static void Do(object source, params CaseInfo[] cases) {
        var type = source.GetType();
        foreach (var entry in cases) {
            if (entry.IsDefault || type == entry.Target) {
                entry.Action(source);
                break;
            }
        }
    }

    public static CaseInfo Case<T>(Action action) {
        return new CaseInfo() {
            Action = x => action(),
            Target = typeof(T)
        };
    }

    public static CaseInfo Case<T>(Action<T> action) {
        return new CaseInfo() {
            Action = (x) => action((T)x),
            Target = typeof(T)
        };
    }

    public static CaseInfo Default(Action action) {
        return new CaseInfo() {
            Action = x => action(),
            IsDefault = true
        };
    }
}

Using live mesh to synchronize favorites

2008-05-15T00:00:00+00:00

I’m a huge fan of customizing my environment. As a developer my productivity is tied to access to my favorite tools, documentation, scripts, plug-ins and generally being happy with the look and feel of my computer. This runs against me using a lot of computers in my job and at home. I spend a lot of time writing scripts to keep my computers in good developer order by manually synchronizing, installing tools, etc …

This one reason I am a huge fan of Live Mesh. It automatically synchronizes files between multiple PC’s allowing me to forgot about annoying web document storage,?? flaky Internet connections, etc.’? This also means that any technology where configuration is based on files/folders can use Live Mesh to synchronize configuration between machines.

Today I’m going to take a small departure from my normal technical blather and explain how you can use this technology in order to make Internet Explorer have the same favorites on all of the machines you run Live Mesh on. Favorites are stored as shortcuts in a folder on your hard drive: namely c:\users\yourusername\Favorites. The first step is to get this folder loaded into Live Mesh. Navigate to c:\users\yourusername, right click on Favorites and select “Add folder to your live Mesh”

The folder will quickly upload into the Mesh. It will upload all of the favorites from the machine where these actions took place.

Now go to another machine where you have Live Mesh installed. There will now be a folder shortcut on your desktop named “Favorites”. Click on that folder and it will bring up the Live Mesh synchronization dialog. Choose the location of favorites on the current machine. It will still likely be c:\users\yourusername\Favorites.

Click Ok and it will bring up a warning saying it will merge the files in the Favorites folder with the files on Live Mesh. Generally speaking this is OK but I advise you to backup the folder first just in case. It will take Live Mesh a few seconds to download the favorites but afterwards IE on both machines will have the same favorites.

Adding, deleting and reordering favorites on one machine will now automatically show up on all machines where you performed this install. Note, I’ve had to restart IE occasionally to get it to recognize the newly added favorites but this is a minor annoyance compared with getting favorites to be the same everywhere.

Reserved Words Good For Your Sanity

2008-05-13T00:00:00+00:00

Paul Vick posted a recent entry exploring the necessity, or lack there of, for having reserved words in a programming language. It’s an interesting mental exercise to go through. At the end you’ll realize that many reserved keywords aren’t needed from the perspective of the compiler. This is part of the reason C# and VB are defaulting more to contextual keywords in recent releases. What the blog post didn’t really talk about though was the programmer. From the programmer’s perspective there is one great reason for reserved words.

Sanity.

Going through legacy code is hard enough already. Reserved words at least allow you to mentally structure the code you are looking at. If there was open season on the use of keywords you know someone will take advantage and flat out abuse the system. Can you imagine digging through some legacy C++ and seeing the following [^1]

class int : public new { public: new private; }
new void() {
  int int = new new;
  int.private = int;
  return int;
}

Reading that makes my head hurt.

Having reserved words is a language adds a modicum of structure. Language structure allows developers to more quickly grasp the meaning and intent of a piece of code. Can you imagine trying to grasp a more complex example? Now imagine the programmer had several thousand lines of undocumented code with this pattern. Not fun.

[^1] Yes I realize that making new/int non-reserved words may not be possible in C++. But this is just a hypothetical example.

Equality Isn T Easy

2008-05-12T00:00:00+00:00

After my recent postings on the rules of Equality, I thought it would be a good idea to post a simple example of equality. The class in question, Example, has only one field of type Integer name m_field1. Two instances of Example are equal if m_field1 has the same value. So the real equality check is just a single Integer comparison.

Unfortunately, as my posts alluded to, even though the check is simple getting it right is not necessarily so. The equality portion of example takes roughly 20 lines of code while the actual equality check represents only 1 of those lines ¹. Not a good ratio. The good and bad news about the other 19 lines is they are boiler plate so once you know them you don’t have to think about them. For my own purposes I’ve converted those 19 lines into a snippet which automates the process but doesn’t make it any easier on the eye.

Class Example
    Implements IEquatable(Of Example)

    Private ReadOnly m_field1 As Integer
    Public Sub New(ByVal field As Integer)
        m_field1 = field
    End Sub

    Public Function Equals1(ByVal other As Example) As Boolean Implements System.IEquatable(Of Example).Equals
        If other Is Nothing Then
            Return False
        End If

        Return m_field1 = other.m_field1
    End Function

    Public Overrides Function Equals(ByVal obj As Object) As Boolean
        Return Equals1(TryCast(obj, Example))
    End Function

    Public Overrides Function GetHashCode() As Integer
        Return m_field1.GetHashCode
    End Function

    Public Shared Operator =(ByVal left As Example, ByVal right As Example) As Boolean
        Return EqualityComparer(Of Example).Default.Equals(left, right)
    End Operator

    Public Shared Operator <>(ByVal left As Example, ByVal right As Example) As Boolean
        Return Not EqualityComparer(Of Example).Default.Equals(left, right)
    End Operator

End Class

Module Module1

    Sub AssertTrue(ByVal cond As Boolean)
        Debug.Assert(cond, "failure")
    End Sub

    Sub AssertFalse(ByVal cond As Boolean)
        Debug.Assert(Not cond, "failure")
    End Sub

    Sub Main()
        Dim v1 As New Example(1)
        Dim v2 As New Example(2)
        Dim v3 As New Example(1)

        AssertFalse(v1 Is v2)
        AssertFalse(v1 Is v3)
        AssertTrue(v1 Is v1)
        AssertFalse(v1 = v2)
        AssertTrue(v1 = v3)
        AssertFalse(v1 = Nothing)
        AssertFalse(Nothing = v1)
        AssertTrue(v1 <> v2)
        AssertTrue(v1 <> Nothing)
        AssertTrue(EqualityComparer(Of Example).Default.Equals(v1, v3))
        AssertFalse(EqualityComparer(Of Example).Default.Equals(v1, v2))
    End Sub

End Module

Before VB criticism enters, C# has roughly the same ratio for the same sample. ↩

Iequatable Of T And Gethashcode

2008-05-09T00:00:00+00:00

This is a bit of a follow up to a previous post we discussed how to properly implement equality in VB. Several users commented/asked that IEquatable(Of T) could be used in place of overriding Equals(). Since IEquatable(Of T) doesn’t define a GetHashCode() method the user didn’t need to define it and hence run into all of the problems associated with GetHashCode() usage.

Unfortunately this is not the case. Several parts of the framework link IEquatable(Of T).Equals and Object.GetHashCode() in the same way that Object.Equals() and Object.GetHashCode() are linked.

The prime example of this is EqualityComparer(Of T). This class is used to provide instances of IEqualityComparer(Of T) for any given type. Under the hood it tries to determine the best way checking for equality in types. If T implements IEquatable(Of T) it will eventually create an instance of GenericEqualityComparer(Of T) ¹. The methods of IEqualityComparer(Of T) are implemented as follows (boundary cases removed)

Equals(left, right): return left.Equals(right)
- Equals in this case is IEquatable(Of T).Equals
GetHashCode(obj): return obj.GetHashCode()

This implicitly links IEquatable(Of T).Equals to Object.GetHashCode(). Therefore if you implement IEquatable(Of T) you should also override GetHashCode(). The best way to view this is “if you implement IEquatable(Of T) you should override Equals and GetHashCode.”

What’s even more unfortunate is there is no feedback to indicate this relationship exists. If you override Equals or GetHashCode both the C# and VB compilers will issue a warning/error. Implementing IEquatable(Of T) produces no such warning.

Unfortunately this is a private type so you will need to use Reflector to view the type. ↩

Saved By Powershell

2008-05-07T00:00:00+00:00

Recently I made a very large update to our code base. Our code base lacked a standard way of guarding entry and exit points into the various components. Having said guards is useful for error handling, tracing, reducing redundancy, etc … The edit standardized our entry points by adding start/end macros to our entry point functions. In addition to other house keeping, the macros also created an HRESULT variable named “hr”. Example below.

    #define MY_ENTRY_GUARD() HRESULT hr
    #define MY_ENTRY_EXIT() return hr

Ran suites, everything passed, checked in. Then I got an email from another dev who spent some time tracking down a bug related to this check-in (sorry Calvin). He discovered one scenario my fix did not take into account.

SomeMethod
{
  MY_ENTRY_GUARD();
  if ( somecondition ) {
    HRESULT hr = E_FAIL; // shadows the first hr
  }
  MY_ENTRY_EXIT(); // returns unmodified hr
}

The double declaration of the variable “hr” is not an error or even a warning in C++. Instead the inner “hr” shadows the outer and hence the rest of the method doesn’t update the “hr” which is actually returned. So now I had to find every place in this change where a nested hr was declared. Did I mention this edit was huge’ Going through by hand would not only be time consuming, it would also be very error prone.

At first I considered parsing out the C++ and doing basic brace matching to look for shadowing “hr” variables. I ruled that out due to the amount of time I would need to invest in the script to take into account comments, string literals, etc … Really I didn’t need brace matching, I really just needed to know when I entered and left a method. Almost all C++ methods have their opening and closing braces on the first column. Writing a script to detect this is trivial.

Script took about 5 minutes to write and 10 to run in the code base. Saved me countless hours of error prone reviews. Thank you PowerShell.

Find-DoubleHr.ps1:

param ( $argFileName = $(throw "Need a file name") )

function Do-Work() {  
   $i = 0  
   foreach ( $line in (gc $argFileName) )   {  
       new-tuple "Text",$line,"LineNumber",$i  
       $i++  
   }  
}

function Do-Parse() {  
   begin {  
       $inMethod = $false  
       $seenHresult = 0  
       $seenMacro = 0  
   }  
   process {  
       $tuple = $_  
       if ( $inMethod ) {  
        switch -regex ($tuple.Text) {  
           "^}" {  
               $inMethod = $false  
               if ( ($seenHresult -ne 0 )-and ($seenMacro -ne 0) ) {  
                "Found a double {0},{1}" -f $seenHResult,$seenMacro  
               }  
               $seenHresult = 0  
               $seenMacro = 0  
               break  
           }  
           ".*MY_ENTRY_GUARD.*" {  
               write-debug ("Macro: {0} " -f $tuple.Text)  
               $seenMacro = $tuple.LineNumber  
               break  
           }  
           "HRESULT.*\Whr\W" {  
               write-debug ("HResult: {0}" -f $tuple.Text)  
               $seenHresult = $tuple.LineNumber  
               break  
           }  
        }  
       }  
       elseif ( $tuple.Text -match "^{" ) {  
        $inMethod = $true  
       }  
   }  
}

"Processing $argFileName"  
Do-Work | Do-Parse

C Programming Books

2008-05-01T00:00:00+00:00

I was reading a post on Coding Horror the other day about programming books and how developers don’t read enough of them.?? I readily agree with the first two points in the article that 1) most programming books suck and 2) books are sold by weight not by volume. Reading is an integral part of a developers life. Blogs are a great source of info but books are a necessity as well.

In my experience, the hardest language to find a good book for is C++. Far too many books and far too few which are worth reading. Here are my favorite which 1) don’t suck and 2) don’t weigh a ton but have a enormous amount of information.

Herb Sutter: 1. Exceptional C++ 2. More Exceptional C++ 3. Exceptional C++ Style
Scott Meyers 1. Effective C++ 2. More Effective C++

Others may disagree but I classify the Scott Meyer books as being good for developers looking to break into the intermediate level. Herb Sutter is great for developers looking to cross from a intermediate to efficient level in C++.

For developers who think they’re ready to cross into the expert realm I highly recommend “C++ Template Metaprogramming.”?? It should convince you that being an expert it C++ is near impossible :)

IMHO, the first two Herb Sutter books I listed should be required reading for anyone doing C++ software development.

Rantpack A Utility Library

2008-04-30T00:00:00+00:00

I often post code examples, samples and snippets on this blog. Many of these samples are a part of a utility library I’ve been writing and maintaining for many years now. Essentially since I got involved in DotNet programming.

I write a lot of code for internal apps, demos and hobby projects. Having a utility library to tote around with me greatly increases my impact in a project because it gives me a place to put common code that I would otherwise have to repeatedly cut and paste into projects.

Whenever I work on something I think the community at large would benefit from I try to blog about it. Unfortunately for the more complex samples blogging is not always easy. Complex structures don’t really fit on a blog and often utilities depend on utilities and it forces users to cut and paste between multiple blog entries. Not good.

Now with http://code.msdn.com I have a forum to fully share the items I’ve been working on. This is the perfect place to share code for the community and helps avoid the messiness of cutting and pasting blog posts. As such I’ve released my Library, RantPack, on Code Gallery. This includes both binaries and sources and is covered under the Microsoft Public License.

RantPack is available at http://code.msdn.microsoft.com/RantPack

A couple of notes on the sample. This is mainly for education and sharing purposes. The goal of this library is to make developing software faster and more reliable. It includes, but is not limited to

Functional Programming Patterns
- Tuples
- Immutable/Persistent collections
Threading
- Futures
- Active Objects
- Various other primitives
General Patterns
General Utilities

Another item you will find browsing the sources are tests. Lots and lots of tests. At the time of this writing there are around 400 some odd unit tests on the library. There is nothing more frustrating than a utility library that doesn’t work. As such I rigorously test the code I write.

Although I performance test much of my code and it performs great in the applications I use it for I’m sure there are a few cases I missed. Please feel free to point this out to me and I’ll look into them.

Why the name RantPack? My blog is Rantings of a programmer and it’s a pack of code :)

Properly Implementing Equality In Vb

2008-04-28T00:00:00+00:00

Many developers want to implement equality functions for their objects. DotNet made equality a deep part of the framework and added support all the way up to System.Object with Equals and GetHashCode.’? In addition to the strongly enforced method semantics of GetHashCode and Equals there are also several other hard to enforce patterns that developers must follow in order to properly integrate into the rest of the DotNet framework. We’ll explore those rules today.

Before even talking about how to implement equality we need to define the types of equality.

Reference equality: Are these two objects really the exact same object.
Object/value equality: Depends on what the object author thinks equality means. Can be anything from reference equality up to, comparing fields, to were they created in the same app domain.

In VB these are two very separate types of equality. Reference equality is expressed through the “Is” operator. Object equality is done directly through operator=, operator<> and Equals. This is also indirectly exposed via GetHashCode, EqualityComparer(Of T) and other framework patterns.

When implementing object/value Equality there are four methods that are important to consider in order to fit expected patterns. What’s even more important is understanding which ones must be implemented together. If an author overrides any of the functions in the following pairs they **must **override both.

Equals/GetHashCode
Operator=/Operator!=

To make it easier, my rule of thumb is to override all four or none.

Equals

This is the bread and butter of object/value equality. The author has free reign to decide what is and what is not equal. However there are a few rules authors must follow in order to fit into the rest of the framework.

Do not throw an exception from Equals. Many components call Equals in a loop and there is no way for them to handle or recover from an exception. If the object is not equal just return False
The object passed in is typed to object. It is perfectly valid for the framework to pass in an object that is completely unrelated to the type defining Equals. The type author must account for and handle this case.
The framework can pass in Nothing as a parameter to Equals and this is valid.

#2 and #3 may seem a bit off at first but it is implemented with a standard pattern as seen below.

Class C1
    Public Overrides Function Equals(ByVal obj As Object) As Boolean
        Dim other = TryCast(obj, C1)
        If other Is Nothing Then
            Return False
        End If
        ...
    End Function
End Class

It’s very important that you use “Is” to compare other in the above example. Imagine if you slip up and type “=” instead. You’re about to override Operator= and this will cause “other=Nothing” to call operator=. If this is a valid C1 instance operator= will almost certainly call Equals and then you’d have a stack overflow. Our implementation of Operator= below will avoid this problem.

GetHashCode

This is both the easiest and trickiest function to override because it has very subtle semantics which cause very hard to find bugs in code. The simple rule is “If two objects are equal in the sense of value equality they must return the same value in GetHashCode()”.

Why’ Many classes use the hash code to classify an object. In particular hash tables and dictionaries tend to place objects in buckets based on their hash code. When checking if an object is already in the hash table it will first look for it in a bucket. If two objects are equal but have different hash codes they may be put into different buckets and the dictionary would fail to lookup the object.

The better version of the GetHashCode rule has a small suffix on the simple rule. “Only calculate the hash code based off of primitive fields which are ReadOnly”. This is not an absolute requirement as long as you are careful when you are code. But as previously stated, when coding it’s best not to trust yourself to do the right thing. Not doing this will get you into trouble when dealing with Hashtables and dictionaries.

For instance take this not so small example. In this case value equality is based solely off of Field1 which is a modifiable field. Once Field1 is changed you may or may not be able to access the value in the dictionary because GetHashCode() will change. This example is contrived but it does happen in the real world and it can be incredibly difficult to track down.

Class C2
    Public Field1 As Integer

    Public Sub New(ByVal f1 As Integer)
        Field1 = f1
    End Sub
    Public Overrides Function GetHashCode() As Integer
        Return Field1.GetHashCode()
    End Function
    Public Overrides Function Equals(ByVal obj As Object) As Boolean
        Dim other = TryCast(obj, C2)
        If other Is Nothing Then
            Return False
        End If
        Return other.Field1 = Field1
    End Function
End Class

Module Module1

    Sub Main()
        Dim map = New Dictionary(Of C2, String)
        Dim v1 = New C2(44)
        map.Add(v1, "avalue")
        Console.WriteLine(map(v1))
        v1.Field1 = 2
        Console.WriteLine(map(v1))  ' Potentially throws
    End Sub

End Module

If Field1 were ReadOnly there would be no way to hit this problem. Then again we’d also not be able to change Field1.

Operator=

When implementing equality overriding operator= allows you to use the more pleasant and reliable version of syntax comparison: a=b vs. a.Equals(b). I say more reliable because using a.Equals(b) has an inherent dependency on “a” being a non-Nothing object. “Operator=” makes no assumption and should operate correctly in the presence of Nothing.

Operator= has virtually the same rules as Equals. Mainly don’t throw from operator=. Operator= is usually just defined in terms of Equals() and since it also has to respect the no throw rule once we get there we are in good shape. Getting to Equals() can be tricky though because one or both of the arguments can be Nothing. In addition make sure not to use “=” to check for Nothing because you’re back to the stack overflow problem.

What’s great here is there is a simple solution that you should use every time you define Operator=. EqualityComparer(Of T) knows all of these rules and in the face of both parameters being non-Nothing will call Equals() just like we want. This makes the definition of Operator= boiler plate (I define very Operator= the exact same way)

Public Shared Operator =(ByVal left As C2, ByVal right As C2) As Boolean
    Return EqualityComparer(Of C2).Default.Equals(left, right)
End Operator

What’s even better is that EqualityComparer(Of T) understands the stack overflow problem which can occur in equality comparison and avoids it.

Operator <>

Operator<> has the same rules as Operator= and luckily the same easy type of answer.

Public Shared Operator <>(ByVal left As C2, ByVal right As C2) As Boolean
    Return Not EqualityComparer(Of C2).Default.Equals(left, right)
End Operator

Wrapping Up

I started this article thinking it would be a few paragraphs of simple rules. But as I kept going I kept remembering the subtleties and problems I encountered in the past.

For my own projects I avoid implementing equality unless it’s truly needed because of the problems with properly implementing GetHashCode(). The one exception is when I define immutable objects. Immutable objects have no problems with GetHashCode() since they are unchangable so Equality is straight forward.

Me Mybase Myclass And Mypost On The Subject

2008-04-25T00:00:00+00:00

Recently we had a good discussion on an internal alias about the use of Me, MyClass and MyBase in VB. Me, MyBase and MyClass are all ways to access instance member data in a VB class or structure. There was a little bit of confusion on the actual workings and meanings of the keywords in various contexts and I want to use this post to shed light on the different meanings.

The Basics

The keywords are used to alter the way in which instance members of a class/structure are accessed. In particular they affect the way Overridable/MustOverride/Overrides functions are evaluated. Methods defined with Overridable/Overrides/MustOverride are defined as virtual by the CLR. For the purpose of this post all of these definitions are mostly equal. This discussion will be useless without and example so here’s the code to discuss.

Class GrandParent
    Public Overridable Sub Sub1()
        Console.WriteLine("GrandParent.Sub1")
    End Sub
End Class

Class Parent
    Inherits GrandParent
    Public Overrides Sub Sub1()
        Console.WriteLine("Parent.Sub1")
    End Sub
End Class

Class Child
    Inherits Parent
    Public Overrides Sub Sub1()
        Console.WriteLine("Child.Sub1")
    End Sub
End Class

In this case Sub1 is a virtual method and there are three instances of it (one per class). By default virtual methods are called based on the runtime type of the object. The CLR will essentially walk the hierarchy from current type to object looking for the first class which defines a particular method and call that version. It doesn’t matter what the variable type is declared as, just what type it actually is.

Dim v1 As GrandParent = New Parent
v1.Sub1()   ' Calls Parent.Sub1
Dim v2 As GrandParent = New GrandParent
v2.Sub1()   ' Calls GrandParent.Sub1

Changing the call

If the CLR will always call a virtual Sub/Function based on the runtime type of an object how can I access the parent function? This is where MyBase comes in. MyBase allows you to call the version of the virtual method defined in the parent class. Essentially it tells the CLR call this method/property as if my runtime type was my base type.

Class Child2
    Inherits Parent
    Public Overrides Sub Sub1()
        MyBase.Sub1()   ' Calls Parent.Sub1
        Console.WriteLine("Child2.Sub1")
    End Sub
End Class

MyClass is similar to MyBase. Instead of telling the CLR the current type is the base type, it tells the CLR the runtime type is the type where MyClass is used. This allows developers to call their type’s version of a virtual method no matter who derives from them. In the following example it doesn’t matter how many, or who derives from Child3, Sub2 will always call the version of Sub1 defined in Child3.

Class Child3
    Inherits Parent
    Public Overrides Sub Sub1()
        Console.WriteLine("Child3.Sub1")
    End Sub
    Public Sub Sub2()
        MyClass.Sub1()
    End Sub
End Class

So, why not MyChild?

The short answer is, it’s not verifiable. When writing MyBase we can verify that indeed there is a sub/function/property matching the call site in the base class. If no such method exists it will result in an error. MyClass is similarly easy to verify. With MyChild however there would be no useful way of guaranteeing a particular sub/function/property was defined on the child class.

One way you can verify a child class contains a particular property/sub/function is to make it MustOverride. However in this case there is no actual definition in the original type. In fact, if you try and access a MustOverride method with MyClass it will generate a compile time error. Therefore every call must at least occur in the child class or lower rendering MyChild superfluous.

What about non-virtual methods and properties?

The primary intent of MyBase/MyClass is to call virtual methods in a non- virtual way. However they can also be used to call non-virtual methods. From the perspective of the user calling a non-virtual method with MyBase/MyClass/Me has no discernable difference. If you crack open the generated IL you can see a small difference in the op code but the short story is it won’t affect your program.

Gotcha Ccomautocriticalsection And Copy Constructors

2008-04-24T00:00:00+00:00

While investigating a crash during a suite run I found the stack walk included the destructor for a CComAutoCriticalSection. This is a fairly reliable class so I immediately suspected my code. I did a couple of quick checks for a double free and didn’t find any. Then I looked a little closer at CComAutoCriticalSection and spotted that it doesn’t redefine the standard copy and operator= constructor.

This is a red flag for any class that implements RAII. C++ will automatically define a copy/operator= for your clasess which will amount to a memcpy. This means copies of two objects will be managing the same resource. Once the second one is destroyed it will result in a double free and hopefully a crash.

I walked the hierarchy and discovered that none of the base classes redefined these methods either. After that it only took a few minutes to discover the bug. I attempt to pass the object by reference but instead ended up passing it by value.

To ensure I didn’t miss any other cases of this, I added the following definition to my code base and moved all instances of CComAutoCriticalSection to point to this version.

class SafeCriticalSection : public CComAutoCriticalSection
{
public:
    SafeCriticalSection() {}
private:
    SafeCriticalSection(const SafeCriticalSection&);
    SafeCriticalSection& operator=(const SafeCriticalSection&);
};

It should be standard practice to define all 3 of the above methods on any class which has RAII semantics. This is not necessary if all of the members are copy safe (CComPtr<> for example).

Get Tfstatus

2008-04-23T00:00:00+00:00

Been far too long since I blogged about a new PowerShell script. This is not to say I’ve stopped using PowerShell, more that I’ve been too busy playing with other tools to spend a significant amount of time updating my scripts.

This is a simple, yet straight forward script. The intent is to make the “tf status” command easier to use from PowerShell. I often want to do some last second verification on files I’ve altered. The default output is not easy to one time parse so a script is handy.

function Get-TfStatus() {  
    param ( [string]$path= "." ,  
            [switch]$recursive = $false ) 

    $args = ""  
    if ( $recursive ) {  
        $args = "/r"  
    }  
    $output = [string[]](& tf status $path $args) 

    # First two lines are junk so skip past it  
    for ( $i = 2; $i -lt $output.Length; $i++ ) {  
        $name,$edit,$path = $output[$i].Split(" ", [StringSplitOptions]"RemoveEmptyEntries")  
        if ( $path -and (test-path $path) ) {  
            new-tuple "FileName",$name,"Change",$edit,"FilePath",$path  
        }  
    }  
}

Which has the output

$PS> get-tfstatus -r

FileName                   Change                     FilePath  
\--------                   \------                     \--------  
File1.cpp               edit                       E:\dd\sourcepath\src\v...  
File2.cpp               edit                       E:\dd\sourcepath\src\v...  
File3.h                  edit                       E:\dd\sourcepath\src\v...

Api Design Readonlycollection T

2008-04-22T00:00:00+00:00

I am a huge fan of read only/immutable collections and data. Hopefully the increased exposure through the blogosphere alerted users to the advantages of this type of programming for the appropriate scenarios. I wanted to discuss [ReadOnlyCollection](http://msdn2.microsoft.com/en- us/library/ms132474.aspx) in case devs looking around in the BCL discover it and assume it's immutable. There are two details of this class which cause gotchas and design issues for consumers who assume it is immutable.

It implements IList

[IList](http://msdn2.microsoft.com/en-us/library/5y536ey6.aspx) is a interface describing mutable collection types which support indexing. ReadOnlyCollection is designed to be read only and cannot fulfill this contract. Therfore every mutable function will throw an exception. IMHO this is not the best design because it is implementing a contract it won't every fullfill. This has the effect of turning what should be a compile time error into a runtime exception (passing a non-mutable collection to an API expecting a mutable collection).

Unfortunately there is not a good interface to implement. The indexable interfaces are all representative of mutable collection types. It would be nice to add an immutable/read only interface which can be safely implemented.

interface IReadOnlyList<T> : IEnumerable<T>
{
    T this[int index] { get; }
    int Count { get; }
    int IndexOf(T value);
    bool Contains(T item);
    void CopyTo(T[] array, int arrayIndex);
}

Ideally this would be called IImmutableList but I'm having trouble getting over the double I, double M pattern to start the name. Perhaps IPersistentList.

It’s not deeply ReadOnly

ReadOnlyCollection is a read only facade on top of mutable collection. Read only data should be just that, read only. This means calling methods directly on the class produce the same result every single time. However because it uses a mutable backing store this is not true and can cause gotchas along the road.

Take the following sample which uses a ReadOnlyCollection to wrap a List.

var list = new List<int>();
list.AddRange(Enumerable.Range(1, 10));

var roList = new ReadOnlyCollection<int>(list);
Console.WriteLine(roList.Count);    // Outputs: 10
list.Add(42);
Console.WriteLine(roList.Count);    // Outputs: 11

There are ways to avoid this problem with ReadOnlyCollectionn. The simplest is to make sure you pass a copy of your list into the constructor.

var roList2 = new ReadOnlyCollection<int>(new List<int>(list));

Thread Local Storage Template

2008-04-21T00:00:00+00:00

Thread local storage is another method of synchronization between threads. It is different that most synchronization cases because instead of sharing state between threads it enables developers to have independent, thread specific pieces of data which have a similar or common purpose.

The uses of thread local storage (TLS) vary greatly but is a very powerful and lightweight method for storing data. TLS can easily be envisioned as a giant void* array for every thread. The entry point, TlsAlloc, provides an index into this array and allows the storage of arbitrary data ¹.

TLS is particularly useful for storing state information. For example, one of my components lives in a highly multi-threaded environment. Each thread serves essentially the same purpose and has the same states and state transition semantics. Like any good paranoid programmer I wanted to add contracts to check my state transitions and semantics.

Contract.VerifyState(ExpectedState, ‘CurrentState)

The question is where to store the state information for a thread? A global state variable won’t suffice because there are N threads. A global array of state information also has it’s share of problems: synchronization, determining an index, lifetime.

TLS is ideally suited to this scenario. Each thread has an independent but similar concept of state. In my initialization code I allocate an TLS index and now I have a place to store my state.

Contract.VerifyState(ExpectedState, *TlsGetValue(g_stateTlsIndex)

The next question is how to manage the lifetime? TLS provides a void* and the caller must manage the lifetime of the allocated memory. Since this is thread specific the ideal place is to manage the memory in the thread startup proc. However I don’t own the creation of the thread, my component is called on a number of threads so this won’t work.

The solution is to use the stack. The initial return for TlsGetValue is NULL. If this situation is detected then the current stack frame is set to own the memory for the slot. Further accesses to the value do not own the memory and simply access it. The semantics are straight forward but annoying to constantly rewrite, so naturally write a template :)

template <typename T>
class TlsValue
{
public:
    TlsValue(DWORD index, const T& defaultValue=T()) :
        m_pValue(NULL),
        m_index(index),
        m_owns(false)
    {
        m_pValue = reinterpret_cast<T*>(::TlsGetValue(m_index));
        if ( !m_pValue )
        {
            m_pValue = new T(defaultValue);
            m_owns = true;
            ::TlsSetValue(m_index, m_pValue);
        }
    }
    ~TlsValue()
    {
        if ( m_owns )
        {
            ::TlsSetValue(m_index, NULL);
            delete m_pValue;
        }
    }

    T* Value() const
    {
        return m_pValue;
    }

private:
    // Do not auto generate
    TlsValue();
    TlsValue(const TlsValue<T>&);
    TlsValue& operator=(const TlsValue<T>&);

    T* m_pValue;
    DWORD m_index;
    bool m_owns;
};

In addition to this blog post, I added a working sample to http://code.msdn.microsoft.com/TlsValue. This is my first attempt at posting a sample on http://code.msdn.com so please provide any and all feedback on the data.

This is similar to data marked with the ThreadStatic attribute in managed code without all of the slot messiness and with the added benefit of strong typing. ↩

What's the purpose of this blog?

2008-04-17T00:00:00+00:00

I’ve had a couple people ask me this question about my blog. The simple answer is: to explore my adventures in code, coding, patterns and pretty much anything else related to programming.

I realize from a readers perspective my topics may appear somewhat random. After all I post in C#,VB,C++ and Powershell with varying degrees of frequency. I jump from patterns to specialized templates to threading to random gotchas to API Design. The reason for the jumping is I post topics close to the projects I’m working on and I work on a lot of projects :). At any given time I’m working on one of the following items.

My job as developer on the VB Debugger and IDE
My personal managed utility libraries (non-UI and UI) ¹
The PInvoke Interop Assistant
Several internal tools
My scripting environment
Customer reported issues in forums
This blog
Whatever happens to capture my interest at the time.

Whenever I discover anything interesting, find a bug or see yet another pattern/design I like or dislike I try to blog about it. I do this for two reasons.

My own personal sanity. I’ve found through years of programming that it’s very unlikely you will hit an error only once. Instead you’re likely to see it time and time again. Blogging about it both helps firm the issue into memory and gives me a great reference to fall back onto the next time I see that bug. It’s really simple to lookup a blog post by tag and read it or point a coworker to it.
Everyone else’s sanity. The more people who write about issues, the more likely someone else will find an answer when they enter a problem into a search engine.

In short, I love to write and talk about code and this is a great forum.

Hopefully on it’s way to http://code.msdn.com so I don’t have to keep posting little snippets and making users paste snippets from various blog entries to get a simple working example. ↩

Gotcha Generic Overload Resolution When Called Generically

2008-04-14T00:00:00+00:00

Both VB and C# have a feature of generic overload resolution that is fairly helpful and yet a source of gotchas. Lets say you have two methods with the same number of arguments. One method has arguments with generic types and the other does not. For Example:

class C1<T> {
    public void F1(int ival) {
        Console.WriteLine("Non-generic F1");
    }
    public void F1(T val) {
        Console.WriteLine("Generic F1");
    }
}

Imagine what happens when we have a C1 and call F1(5). Which method should the compiler bind to? Both VB and C# when presented with this type of situation will choose the non-generic version.

v1.F1(5);           // Binds to Non-generic F1
v2.F1("foo");       // Binds to Generic F1

At a glance we might think that we can provide specialized behavior for a subset of types that are interesting (similar to C++ template specialization). If VB/C# will bind to our non-generic version then we don’t have to due ugly type switching to implement different behavior for certain types. Unfortunately we would be wrong.

This type of overload resolution only happens if the compiler can statically verify that the argument matches a non-generic overload. So if we call F1 generically it will not bind to the non-generic overload.

static void TestOverload<T>(C1<T> weird, T value) {
    weird.F1(value);
}

static void Main(string[] args) {
    var v1 = new C1<int>();
    var v2 = new C1<string>();

    TestOverload(v1, 5);        // Binds to Generic F1
    TestOverload(v2, "foo");    // Binds to Generic F1
}

Design Guidelines Provide Type Inference Friendly Create Function For Generic Objects

2008-04-11T00:00:00+00:00

Really this guideline is a bit longer but putting it all in a blog title seemed a bit too much. The full guideline should read: “If a generic class constructor arguments contain types of all generic parameters, provide a static method named Create on a static class of the same class name as the generic class which takes the same arguments and calls the constructor.” Quite a mouth full.

Lets look at a specific example with Tuples. Tuples are generic with respect to the values they are representing. Without any type inference help we would have to write the following code to create a simple tuple.

var tuple = new Tuple<int, string>(5, "astring");

Not too bad because we are using simple types. But what happens when we are using really long type names?

var tuple2 = new Tuple<string,Dictionary<string, List<int>>>(val1, val2);

As we can see, the code is getting quite a bit uglier. This pattern is in fact not maintainable once we start using un-namable types such as anonymous types or generics of anonymous types.

The problem here is we are not leveraging the compilers type inference capabilities. The compiler can easily infer the types of a tuple argument and hence create a tuple. We just need to provide a mechanism to do so. The best way is to define a static method on a static class with the same name as the generic. Lets call this method Create.

public static class Tuple {
    public static Tuple<TA, TB> Create<TA, TB>(TA valueA, TB valueB) { 
        return new Tuple<TA, TB>(valueA, valueB); 
    }
}

The method Tuple.Create still has two generic parameters. However since we are providing a set of values which contain types for every generic parameter, the compiler can infer the generic arguments. Now we can create a Tuple without specifying any generic arguments. Because no types are specified this will work with any value in the code including un-namable types.

var tuple = Tuple.Create(6, "astring");
var tuple2 = Tuple.Create(6, new { name = "aname", value = 42 });

Have An Icomparer T But Need An Icomparable T

2008-04-09T00:00:00+00:00

Previously we discussed the opposite problem. This is a lesser but often more frustrating problem because there is no, AFAIK, built in solution for the BCL. However it’s problem that can be solved once and reused with a generic solution. IComparable has all of the methods necessary implement IComparer.

To work around this we’ll create a new class that can wrap both a value and an instance of IComparable(Of T). Lets call it ComparerNode. This class can be used wherever an IComparable is needed.

Unfortunately generic classes will not provide a 1:1 mapping. However getting to the actual data is strongly typed and comes through a simple property.

public sealed class ComparerNode<T> : IComparable<ComparerNode<T>> {
    private readonly IComparer<T> m_comparer;
    private readonly T m_value;

    public IComparer<T> Comparer {
        get { return m_comparer; }
    }

    public T Value {
        get { return m_value; }
    }

    public ComparerNode(IComparer<T> comparer, T value) {
        if (comparer == null) {
            throw new ArgumentNullException("comparer");
        }

        m_comparer = comparer;
        m_value = value;
    }

    public override int GetHashCode() {
        if (m_value == null) {
            return 0;
        }

        return m_value.GetHashCode();
    }

    public override bool Equals(object obj) {
        var other = obj as ComparerNode<T>;
        if (other == null) {
            return false;
        }

        return 0 == m_comparer.Compare(m_value, other.m_value);
    }

    #region IComparable<ComparerNode<T>> Members

    public int CompareTo(ComparerNode<T> other) {
        if (other == null) {
            return 1;
        }

        return m_comparer.Compare(m_value, other.m_value);
    }

    #endregion

    public static bool operator <(ComparerNode<T> left, ComparerNode<T> right) {
        return Comparer<ComparerNode<T>>.Default.Compare(left, right) < 0;
    }

    public static bool operator >(ComparerNode<T> left, ComparerNode<T> right) {
        return Comparer<ComparerNode<T>>.Default.Compare(left, right) > 0;
    }

    public static bool operator ==(ComparerNode<T> left, ComparerNode<T> right) {
        return EqualityComparer<ComparerNode<T>>.Default.Equals(left, right);
    }

    public static bool operator !=(ComparerNode<T> left, ComparerNode<T> right) {
        return !EqualityComparer<ComparerNode<T>>.Default.Equals(left, right);
    }
}

public static class ComparerNode {
    public static ComparerNode<T> Create<T>(IComparer<T> comparer, T value) {
        return new ComparerNode<T>(comparer, value);
    }

    public static ComparerNode<T> Create<T>(T value)
        where T : IComparable<T> {
        return new ComparerNode<T>(Comparer<T>.Default, value);
    }
}

Gotcha Ccomptrbase T Assignment

2008-04-08T00:00:00+00:00

Today what started out as a crash due to a pure virtual call turned into finding a gotcha in CComPtrBase. Essentially the code in question boiled down to the following. Can you spot the problem?

void GetAStudent(CComPtrBase<T> &spStudent)
{
    CComPtr<Student> spLocal;
    // Do some operation to get a student
    spLocal = spStudent;
}

The problem isn’t apparent until you look at the definition for CComPtrBase::operater =. See the problem? Basically CComPtrBase::operator= isn't explicitly defined. This means that C++ will automatically implement [memberwise assignment.](http://msdn2.microsoft.com/en-us/library/x0c54csc$VS.71$.aspx) The RHS of the operator= will be a "const CComPtrBase&".

CComPtr derives from CComPtrBase therefore it satisfies this and a memberwise assignment will occur. We now have two smart pointers on the same value. However the second smart pointer, CComPtrBase, did not perform an AddRef. So when both objects are destroyed there will be an extra Release and hopefully a crash.

The fix?

Use CComPtr or CComPtrEx instead of CComPtrBase
Less Optimal: call AddRef() on spLocal.

Binaryinsert Part2

2008-04-07T00:00:00+00:00

Previously I discussed a potential missing API in List(Of T).BinaryInsert. One of the items I mentioned was it had better performance because it was O(Log N) vs Insert and Sort which is O(NLogN). Several users correctly pointed out this was incorrect and that Insert() had the additional overhead of an Array.Copy() which is O(N)ish. But most agreed O(N) + O(LogN) was better than O(NLogN).

Given that I already missed a key portion, I decided to write a test program to try out the various methods. Caveat: I’m not a performance guy. While I find performance intriguing and interesting it is by no means my specialty. Any single performance test is unlikely to capture all real world scenarios. However I did find the results a bit surprising. At the bottom of the post is the test code I wrote.

Here is the summary output.

Force Jit  
BinaryInsert:     00:00:00.0051167  
Insert Then Sort: 00:00:00.0000251  
Range (0-99)  
BinaryInsert:     00:00:00.0000266  
Insert Then Sort: 00:00:00.0000316  
Random 10  
BinaryInsert:     00:00:00.0000053  
Insert Then Sort: 00:00:00.0000034  
Random 100  
BinaryInsert:     00:00:00.0000294  
Insert Then Sort: 00:00:00.0000235  
Random 1000  
BinaryInsert:     00:00:00.0004917  
Insert Then Sort: 00:00:00.0001526  
Random 10000  
BinaryInsert:     00:00:00.0261899  
Insert Then Sort: 00:00:00.0018287  
Random 100000  
BinaryInsert:     00:00:02.4289054  
Insert Then Sort: 00:00:00.0237019

As you can see, based on my sample program, BinaryInsert is much slower than Insert and Sort. I ran the profiler against this and verified the suspicion that List(Of T).Insert() took the vast majority of the time.

Perhaps there is a reason BinaryInsert is missing.

Module Module1

    Function BinaryInsert(Of T)(ByVal enumerable As IEnumerable(Of T), ByVal comp As IComparer(Of T)) As TimeSpan
        Dim list As New List(Of T)
        Dim watch As New Stopwatch()

        watch.Start()
        For Each value In enumerable
            list.BinaryInsert(value, comp)
        Next
        watch.Stop()
        Return watch.Elapsed
    End Function

    Function InsertAllThenSort(Of T)(ByVal enumerable As IEnumerable(Of T), ByVal comp As IComparer(Of T)) As TimeSpan
        Dim list As New List(Of T)
        Dim watch As New Stopwatch()

        watch.Start()
        For Each value In enumerable
            list.Add(value)
        Next
        list.Sort(comp)
        watch.Stop()
        Return watch.Elapsed
    End Function

    Sub TestBoth(Of T)(ByVal title As String, ByVal enumerable As IEnumerable(Of T))
        TestBoth(title, enumerable, Comparer(Of T).Default)
    End Sub

    Sub TestBoth(Of T)(ByVal title As String, ByVal enumerable As IEnumerable(Of T), ByVal comp As IComparer(Of T))
        Dim copy = New List(Of T)(enumerable)
        Dim ellapsedBinary = BinaryInsert(New List(Of T)(copy), comp)
        Dim ellapsedSort = InsertAllThenSort(New List(Of T)(copy), comp)

        Console.WriteLine(title)
        Console.WriteLine("BinaryInsert:     {0}", ellapsedBinary)
        Console.WriteLine("Insert Then Sort: {0}", ellapsedSort)
    End Sub

    Function Range(ByVal start As Integer, ByVal count As Integer) As List(Of Integer)
        Dim list = New List(Of Integer)
        For i = start To count - 1
            list.Add(i)
        Next
        Return list
    End Function

    Function Random(ByVal count As Integer) As List(Of Integer)
        Dim rand As New Random()
        Dim list = New List(Of Integer)
        For i = 0 To count - 1
            list.Add(rand.Next())
        Next
        Return list
    End Function

    Sub Main()
        TestBoth("Force Jit", New Integer() {1})
        TestBoth("Range (0-99)", Range(0, 100))
        TestBoth("Random 10", Random(10))
        TestBoth("Random 100", Random(100))
        TestBoth("Random 1000", Random(1000))
        TestBoth("Random 10000", Random(10000))
        TestBoth("Random 100000", Random(100000))
    End Sub

End Module

Reference Values In C

2008-04-03T00:00:00+00:00

Reference values are a powerful feature of C++ but I find they have one significant detractor. A developer can not look at an API call and determine if a parameter is being passed by reference or value (VB has the same problem).

IMHO this is one item that C# got 100% correct. In C# developers must say a value is out/ref or a compile error results. Forcing both the API declaration and usage to specify the reference semantics makes code much more understandable. When you look at an API call there is absolutely no question about the byref/byval/out semantics of a parameter.

Internally I’ve met people who are hesitant to use reference parameters in C++ because of the ambiguity. Not making it declarative in both places meant unexpected behavior could occur in a number of scenarios. I agree with this statement.

But hey we’re talking about C++ here. Any C++ problem can be fixed with some macros and a template right? I thought about this over the weekend and came up with a quick sample. Note, I haven’t extensively tested this sample yet so there may be bugs. However it gets the base cases right.

The goal of this API is to allow API authors to force developers to be explicit about their ByRef semantics. It will prevent developers from silently passing a value by ref and hence getting unexpected behavior. Failure to do so will result in a compile time error. Also there is a minor bit of indirection overhead for debug mode but in retail this will compile out to normal code.

#ifdef DEBUG

template <typename T>
class ByRefType
{
public:
    explicit ByRefType(T& arg) : m_ref(arg)
    {
    }

    operator T&() const 
    {
        return m_ref;
    }

    ByRefType& operator=(const T& value)
    {
        m_ref = value;
        return *this;
    }

private:
    ByRefType();

    mutable T& m_ref;
};

template <typename T>
ByRefType<T> MakeByRefType(T& expr)
{
    return ByRefType<T>(expr);
}

#define ByRef(expr) MakeByRefType(expr)
#define ByRefParam(type) ByRefType<type> 

#else

#define ByRef(expr) expr
#define ByRefParam(type) type&

#endif

All well and good. Now we can attribute byref paramaters with ByRefParam() and force callers to tag it as a ByRef argument.

    void SimpleByRef(ByRefParam(int) i, int newValue)
    {
        i = newValue;
    }
    
    void Test()
    {
        int i1;
        SimpleByRef(ByRef(i1), 5);
        SimpleByRef(i1, 6);     // Compiler Error!!!
    }

As said before, this is an initial implementation and I expect updates as I use this in code and find bugs. Please post back with any you find.

Have A Icomparable Of T But Need An Icomparer Of T

2008-04-02T00:00:00+00:00

IComparable(Of T) is an interface saying “I can compare myself to other objects of the same type”. And IComparer(Of T) is an interface saying “I can compare two objects of this type.”. Often API’s which need to perform comparisons will take an instance of IComparer(Of T) instead of IComparable(Of T).

Doing the opposite is limiting in a few ways. The first is it locks developers into the behavior defined by the type author essentially preventing them from deciding to compare in a different way. The second is if the type author didn’t didn’t implement IComparable(Of T), we must define a wrapper class that does. This is even more awkward because collections now contain instances of the wrapper. Also there is one wrapper overhead per instance in the collection.

Now back to the original problem, we have a type which implements IComparable(Of T) but don’t have an IComparer(Of T) wrapper class. Luckily the .Net Framework provides a solution. Comparer(Of T).Default. It provides a quick IComparer(Of T) wrapper that automatically delegates to IComparable(Of T) if the type defines it.

Example usage: Previously we defined a BinaryInsert method for List(Of T). It required an explicit IComparer(Of T) be passed even for simple types like Int32. We can fix this by using the Comparer(Of T) class.

<Extension()> _
Public Sub BinaryInsert(Of T)(ByVal list As List(Of T), ByVal value As T)
    list.BinaryInsert(value, Comparer(Of T).Default)
End Sub

Short Post

2008-04-01T00:00:00+00:00

In an effort to post more often and with greater effect, I’ve been trying to divide my posts into short, targeted posts and longer in depth posts. Thus far my short posts aren’t nearly as short as I’d like them to be and end up being much closer to my in depth postings in length.

In order to better hit this goal today will be a short post.

Happy April Fools Day!

Missing Api List Of T Binaryinsert

2008-03-31T00:00:00+00:00

One API that seems to be missing from List(Of T) is a BinaryInsert method. Especially since there is already a BinarySearch method.

Binary insert is a method for inserting a value into an already sorted list. Since the list is already sorted we can do a binary search to find the appropriate place to insert. The insert keeps the list sorted so the cost of a binary insert is just the cost of the search which is O(Log(N)).

An alternative method for keeping a sorted list sorted is to insert and then resort. Most sorting algorithms have a cost of O(N*Log(N)). In other words it’s N times more expensive.

Yet this API doesn’t exist. No matter. We can quickly fix this problem with a couple of extension methods.

Public Module Extensions

    <Extension()> _
    Public Sub BinaryInsert(Of T)(ByVal list As List(Of T), ByVal value As T, ByVal comp As IComparer(Of T))
        list.BinaryInsert(value, comp, 0, list.Count)
    End Sub

    <Extension()> _
    Public Sub BinaryInsert(Of T)(ByVal list As List(Of T), _
                                  ByVal value As T, _
                                  ByVal comparer As IComparer(Of T), _
                                  ByVal iStart As Integer, _
                                  ByVal iEnd As Integer)
        While iStart < iEnd
            Dim len = iEnd - iStart
            Dim iMiddle = iStart + (len \ 2)
            Dim comp = comparer.Compare(value, list(iMiddle))
            If 0 = comp Then
                iStart = iMiddle
                Exit While
            ElseIf comp < 0 Then
                iEnd = iMiddle
            Else
                iStart = iMiddle + (len Mod 2)
            End If
        End While
        list.Insert(iStart, value)
    End Sub

End Module

Activeobject

2008-03-30T00:00:00+00:00

I’ve been busy lately and neglected my series on Active Objects. It’s been a fairly busy time for me both in and out of work. Enough excuses, back to the fun.

With the basic PipeSingleReader class, we now have the last piece necessary to create an ActiveObject. This article will focus on building the base ActiveObject which will take care of scheduling, construction, destruction and error handling. The goal is to make implementing an ActiveObject that actually does work easy.

Lets break down the implementation of an ActiveObject into the three phases of any object; construction, destruction and running behavior.

Construction

Active Objects are associated with and have an affinity to a particular thread. Constructing an ActiveObject mainly consists of creating a thread and initializing the member variables of the object.

There are a couple of requirements that need to be met when initializing the object. The first is getting the thread into a known state before returning out of the constructor. It’s possible for coders to create and then immediately destroy an object. Part of destructing an object is understanding the state you are destructing. Returning from an ActiveObject constructor before the thread is up and running means that we can be destructed while in an inconsistent state. Normally this isn’t much an issue with objects because they are single threaded. We will fix this by doing a simple wait until the thread is finished initializing.

protected ActiveObject() {
    m_thread = new Thread(() => InitializeAndRunBackgroundThread());
    m_thread.Start();
    while (0 == m_backgroundInitialized) { Thread.Sleep(0); }
}

Next is providing implementers with a way to initialize member variables on the new thread. There are many reasons for wanting to initialize members on the ActiveObject thread. Besides general consistency concerns, there is also the issue that objects can have affinity to a particular thread and including forcing initialization to occur on that thread. To make this simple part of the thread initialization code will call a virtual method allowing base classes to initialize variables.

private void InitializeAndRunBackgroundThread() {
    Interlocked.Exchange(ref m_affinity, new ThreadAffinity());
    Interlocked.Exchange(ref m_pipe, new PipeSingleReader<Future>());
    InitializeMembersInBackground();
    Interlocked.Exchange(ref m_backgroundInitialized, 1);
    RunBackgroundActions();
}
protected virtual void InitializeMembersInBackground() {
}

Running Behavior

Active Objects exist for one reason, to run Futures. The main behavior is to loop over the set of Futures and run them. The PipeSingleReader class takes care of most of the scheduling and threading work. This leaves the ActiveObject free to make policy decisions.

One question that comes up is how to handle the case where a Future throws an exception’ If we run the Future with no protection it will simple cause an unhandled exception and likely a process crash. We could catch and try to filter them but based on what criteria’ IMHO there is no way to properly handle an exception in the Active Object base because we don’t know what the purpose of that object is. Only the actual object implementer knows. Therefore we will make it their problem by passing unhandled exceptions into an abstract method.

private void RunBackgroundActions() {
    do {
        RunFuture(m_pipe.GetNextOutput());
    } while (0 == m_backgroundFinished);
    Future future;
    while (m_pipe.TryGetOutput(out future)) {
        RunFuture(future);
    }
}
private void RunFuture(Future future) {
    try {
        m_affinity.Check();
        future.Run();
    }
    catch (Exception ex) {
        OnBackgroundUnhandledException(ex);
    }
}

If the second loop looks a bit out of place, hopefully the destruction section will explain it’s purpose.

All that is left is to provide helper methods to let base classes queue up Futures to run.

protected Future RunInBackground(Action action) {
    var f = Future.CreateNoRun(action);
    m_pipe.AddInput(f);
    return f;
}
protected Future<T> RunInBackground<T>(Func<T> func) {
    var f = Future.CreateNoRun(func);
    m_pipe.AddInput(f);
    return f;
}

Destruction

Destruction of an ActiveObject can be tricky with respect to handling pending actions. Should they be executed, aborted or just completely ignored’ What happens if more input is added once we start the dispose process’ If we don’t allow more input where should we error?

IMHO, the simplest user and programming model is the following.

Dispose is synchronous. It will block until the background thread is destroyed. Dispose is the equivalent of destruction so it follows that all resources including the thread will be destroyed when destruction completes.
Once dispose starts input will be stopped. This prevents live-lock scenarios where one thread is disposing the ActiveObject and another thread is constantly adding data.
If another thread tries to add an operation during the middle of disposing they will be given an exception at that time.

In future posts, we’ll explore how to create ActiveObjects with differing dispose semantics.

Now how can we signal the background thread that we are done processing’ Just add a future to the queue to be running. Because this will run on the only thread reading the int there is no need for an Interlocked operation.

private void Dispose(bool disposing) {
    if (disposing) {
        m_pipe.AddInput(Future.CreateNoRun(() => { m_backgroundFinished = 1; }));
        m_pipe.CloseInput();
        m_thread.Join();
    }
}

Now that we’ve gone over the dispose code, hopefully the reason for the second loop in RunBackgroundActions is a little more apparent. Between the two calls to m_pipe in Dispose another thread can post a Future. Without the second loop the user will get no exception and the future will never run. Likely they would hopelessly deadlock.’? The second loop will run all Futures which get caught it this gap.

The Code

Here is the full version of the code.

public abstract class ActiveObject :  IDisposable {
    private PipeSingleReader<Future> m_pipe;
    private ThreadAffinity m_affinity;
    private Thread m_thread;
    private int m_backgroundInitialized;
    private int m_backgroundFinished;

    protected ActiveObject() {
        m_thread = new Thread(() => InitializeAndRunBackgroundThread());
        m_thread.Start();
        while (0 == m_backgroundInitialized) { Thread.Sleep(0); }
    }
    #region Dispose
    public void Dispose() {
        Dispose(true);
        GC.SuppressFinalize(this);
    }
    ~ActiveObject() {
        Dispose(false);
    }
    private void Dispose(bool disposing) {
        if (disposing) {
            m_pipe.AddInput(Future.CreateNoRun(() => { m_backgroundFinished = 1; }));
            m_pipe.CloseInput();
            m_thread.Join();
        }
    }
    #endregion
    private void InitializeAndRunBackgroundThread() {
        Interlocked.Exchange(ref m_affinity, new ThreadAffinity());
        Interlocked.Exchange(ref m_pipe, new PipeSingleReader<Future>());
        InitializeMembersInBackground();
        Interlocked.Exchange(ref m_backgroundInitialized, 1);
        RunBackgroundActions();
    }
    private void RunBackgroundActions() {
        do {
            RunFuture(m_pipe.GetNextOutput());
        } while (0 == m_backgroundFinished);
        Future future;
        while (m_pipe.TryGetOutput(out future)) {
            RunFuture(future);
        }
    }
    private void RunFuture(Future future) {
        try {
            m_affinity.Check();
            future.Run();
        }
        catch (Exception ex) {
            OnBackgroundUnhandledException(ex);
        }
    }
    protected Future RunInBackground(Action action) {
        var f = Future.CreateNoRun(action);
        m_pipe.AddInput(f);
        return f;
    }
    protected Future<T> RunInBackground<T>(Func<T> func) {
        var f = Future.CreateNoRun(func);
        m_pipe.AddInput(f);
        return f;
    }
    protected virtual void InitializeMembersInBackground() {
    }
    protected abstract void OnBackgroundUnhandledException(Exception ex);
}

Api Problems Ccomobject Createinstance

2008-03-28T00:00:00+00:00

CComObject::CreateInstance is a light weight method for creating instances of COM objects in your code. Unfortunately the design of the API makes it easy to introduce subtle errors into your code. The two problems are it encourages manually ref counting and the object initially has a ref count of 0. This means you must remember to AddRef before calling any other function. Neither of these ideas in themselves are bad but it leads to tedious, repetitive code that is too often done incorrectly.

Below is a typical example I see in code.

void AMethod()
{
    CComObject<Student> *pStudent;
    if ( SUCCEEDED(CComObject<Student>::CreateInstance(&pStudent)) )
    {
        pStudent->AddRef();
        VerifyStudent(pStudent);
        pStudent->Release();
    }
}

As a result of the manual ref counting, this code is not exception safe. If VerifyStudent or any API it calls throws, an instance of Student will be leaked.

The second problem I often see in code is the placement of the VerifyStudent function. Occasionally I see methods like VerifyStudent called before the AddRef. If you see this in your code immediately file a bug. The problem is the ref count is 0 before you call AddRef. It COM it should always be legal to AddRef/Release a COM object passed into your function. In this case while legal it will destroy the instance of Student. What’s even worse is this bug can show up years after you code.

Real world example. I created a class to wrap a couple of operations. One of it’s members was Student and as such I wrapped it in a CComPtr<> for ease of use. Fired up the program and everything crashed. Turns out an instance of Student was passed to a function that eventually created an instance of my object before AddRef was called (essentially move VerifyStudent up one line). As soon as my new object died CComPtr<> called Release, moved the RefCount to 0 and destroyed the object.

Writing the correct code is repetitive and begs for a wrapper function. Enter CreateWithRef

template <class T>
static 
HRESULT CreateWithRef(T** ppObject)
{
    CComObject<T> *pObject;
    HRESULT hr = CComObject<T>::CreateInstance(&pObject);
    if ( SUCCEEDED(hr) )
    {
        pObject->AddRef();
        *ppObject = pObject;
    }

    return hr; 
}

void AMethod2()
{
    CComPtr<Student> pStudent;
    if ( SUCCEEDED(CreateWithRef(&pStudent)) )
    {
        VerifyStudent(pStudent);
    }
}

As you can see, using this function takes less typing that a normal CreateInstance due to using type inference. It’s also exception safer since the resource is managed.

This API still has one flaw. It allows people to pass in a raw pointer and hence violate exception safety. It could be improved by forcing a caller to pass in a class which supports RAII. In this case a good choice is CComPtrBase<>. I tend to prefer this design because it forces the caller to use safer code.

template <class T>
static 
HRESULT CreateWithRef2(CComPtrBase<T>& spPointer)
{
    CComObject<T> *pObject;
    HRESULT hr = CComObject<T>::CreateInstance(&pObject);
    if ( SUCCEEDED(hr) )
    {
        pObject->AddRef();
        spPointer.Attach(pObject);
    }

    return hr; 
}

void AMethod3()
{
    CComPtr<Student> pStudent;
    if ( SUCCEEDED(CreateWithRef2(pStudent)) )
    {
        VerifyStudent(pStudent);
    }
}

Part Of Being A Good Programmer Is Learning Not To Trust Yourself

2008-03-24T00:00:00+00:00

… and to actively guard against yourself.

Over the years I’ve found that I can be my own worst enemy when I code. Part of the problem stems from paranoia

Early in my college days, a professor of mine, Jim Greenlee, instilled in me the virtue of paranoid programming. He taught an introduction to C class on a Linux/Unix environment. Our code had to compile with no warnings or errors on Linux and Unix, pass lint, and withstand all manner of evil input the TA’s could throw at it. Catching a mistake and printing out an error was bad, but acceptable. Crashing was an immediate 0. Towards the end of the semester, I usually wrote two programs for the each assignment: 1) the assignment and 2) a Perl script designed to automate and generally speaking cat /dev/random into the program.

Over time, this process led to a healthy suspicion of any code that I used (not so much my code, but code I didn’t own or didn’t actively work in). As a result, I now rarely make assumptions about how other people’s code works. I probe, search, test and investigate any assumption about resource management, lifetime, etc . When I’m working on my code, however, I tend to assume I got it right and followed such patterns as RAII. And therein lies the problem: no one is perfect, and I should be as paranoid about the quality of my code as anyone else’s.

I find this problem happens most often on the border of my code and the rest of the program. I think this is true for most programmers. You understand your model and the manner in which you expect it to behave. On the fringes of your program, you expect other code to behave properly or your code to account for various conditions. In reality, it’s doubtful that both you and the programmer owning the other end of the code had exactly the same thought pattern. Also it’s doubtful you understand all the variations in behavior in the other programmers code.

To further complicate matters, you are not the same programmer you were yesterday. Hopefully you’re better. As a programmer you constantly learn new patterns, better methods and discipline. So when you program against code you’ve written, you’re approaching it with a different mindset and hence you take different paths. In short, you’re programming against someone else’s code and assumptions.

For instance there was a time where I didn’t actively use RAII in my code. Now I base all my C++ classes around this pattern and use it everywhere. These days I tend to assume my classes implement this pattern. I’ve been bitten a few times when using older code that pre-dated my use of RAII.

The best way to avoid making bad assumptions is to actively question them at all times. Guarding against your assumptions can occur at several levels. Over time, I’ve found the following methods to be the most effective:

Don’t trust yourself. B

Be as paranoid about yourself as others.

Turn Assumptions into Compiler Errors

This is most effective when coding in C++. Occasionally I run into a situation where we find certain classes are invalid inputs to a template, function, etc … The best way to prevent using the type in the wrong context is to turn it into a compiler error. Typically this is accomplished with a combination of macros and templates. It’s not pretty, but catching an error at compile time is the cheapest way. You can’t ship a bug that doesn’t compile.

Unit Testing

I once heard an engineer say that “1 test is worth 1000 expert opinions.” It doesn’t matter how good you are or how simple the code is; given enough time, you will make mistakes. The most effective and lasting way to ensure that you don’t make mistakes is to test your code relentlessly.

The other great aspect is it allows you to freeze an assumption in time. The point at which you write the code is the best time to catalog the intended behaviors of your code. Adding a unit test will preserve and enforce it for the perceivable future.

Retail Contracts

Retail contracts are similar to debug asserts. The difference is that they run in retail as well as debug builds. Generally speaking in retail it will cause a crash and Watson dump.

I’ve debated the merits of such an approach many times. But the principal that wins for me is “if it’s wrong at debug time, it’s wrong at retail time”. It’s much easier to catch a problem sooner rather that later. This is especially true for C++ where problems can behave unpredictably and result in “random” failures. Crashing on a line which says Contract.ThrowIfNull() leaves little doubt to where the failure came from. It doesn’t necessarily make finding the underlying cause easier but it’s a start.

I’ve heard the argument that checks are non-performant and shouldn’t be done in retail code. Most retail checks resort down to “if(!ptr) throw”. Yes, if you put many,many of these checks into a hot path of your code base it can cause a performance problem. In my experience it’s very unlikely that this will happen. And if it does show up on a profile, it’s a simple matter to switch them to a debug assert, or move the retail assert further down in the stack and out of the hot path.

Debug Asserts

I find debug asserts are nice for verification that is to expensive to run in retail. I don’t mean this to include simple NULL checks but more deep verification of various data sets.

Code Reviews

I didn’t put this into any specific order because this is highly dependent upon the person reviewing your code. I feel the other listed points add a consistent level of quality to your code. Bad code reviewers offer little help in guarding against yourself and hence do little for quality. Good code reviewers offer good protection but it’s a one time shot so I still would put it between unit testing and retail contracts.

There are the few people who have an amazing talent for code reviews. They will make even the most experience programmer come out feeling like an incompetent novice. They are invaluable and even though they are able to shame me at times I insist upon code reviews from them when writing a feature.

Making Pinvoke Easy

2008-03-14T00:00:00+00:00

I very excited to announce we recently released a tool I’ve been working on to MSDN that will greatly help with using PInvoke in managed code. The tool is called the “PInvoke Interop Assistant” and is included as part of a MSDN article on marshalling data for PInvoke and Reverse PInvoke scenarios.

Here is a link to the article and tool

Article: http://msdn2.microsoft.com/en-us/magazine/cc164193.aspx
Tool: CLRInsideOut2008_01.exe

The motivation behind this tool is writing PInvoke is a hard and often tedious task. There are many rules you must obey and many exceptions that must be taken into account. Anything beyond simple data structures gets very involved and subtle semantics of C can greatly change the needed signature. Incorrect translations often result in obscure exceptions or crashes.

In short, it’s not any fun.

The tool works in several different ways to make PInvoke generation an easier process. The goal is to make generating managed code for structs, unions, enums, constants, functions, typedefs , etc … as easy as possible. The resulting code can be generated in both VB and C#.

The GUI version of the tool operates in 3 modes.

SigImp Search: Search for a commonly used function and translate it into managed code.
SigImp Translate Snippet: Directly translate C code into managed PInvoke signatures.
SigExp: Convert managed binaries into C++ Reverse PInvoke scenarios

The first two are the parts I worked on and represent the PInvoke scenarios. The third part was written by Ladi Prosek and will be covered in a different article. We chose the names SigImp and SigExp to mirror the tblimp/tlbexp tool base since they have similar functions.

Directly translating C code into PInvoke Signatures

Most adventures in PInvoke start with a developer having a small set of C code they would like to use from a managed binary. Typically it’s one or two functions with several supporting C structs. Before, all of this would be hand translated into managed code from scratch. With this tool all you must do is paste the code into the tool and it will generate the interop signature for you.

For instance assume you wanted to translate the following C code into VB.

struct S1
{
  int a;
  char[10] b;
};

float CalculateData(S1* p);

Start up the tool and switch to the “SigImp Translate Snippet” tab. Then paste the code in and then hit the Generate button.

You can also set click the “Auto Generate” box and watch the code update as you type.

This translation is not limited to built-in C types. It will also resolve most commonly used windows types such as HANDLE, DWORD all the way up to complex structs such as WIN32_FIND_DATA

Searching for a commonly used function

Often developers want to use C functions familiar to them in managed code. This can be a tedious task as well because if the signature is not already available you are back to coding from scratch. Even adding a constant value can be tricky if you don’t know which header file to look in.

The tool also provides a database of many commonly used functions, structs, constants, etc … It is essentially anything that is included from windows.h. Switch to the SigImp search tab, type the name of what you are looking for and hit generate. For example if I want to see the value for WM_PAINT just type it in.

In addition this part of the tool will also do dependency calculation. For instance if choose a method which has a parameter that is a C structure it will automatically generate the structure with the function. For instance if you choose the function FindFirstFile it will determine that the function depends on the WIN32_FIND_DATA structure. Furthermore it will notice that WIN32_FIND_DATA depends on FILETIME and generate both in addition to the method.

<System.Runtime.InteropServices.StructLayoutAttribute( _
    System.Runtime.InteropServices.LayoutKind.Sequential, _
    CharSet:=System.Runtime.InteropServices.CharSet.[Unicode])> _
Public Structure WIN32_FIND_DATAW
    '''DWORD->unsigned int
    Public dwFileAttributes As UInteger
    '''FILETIME->_FILETIME
    Public ftCreationTime As FILETIME
    '''FILETIME->_FILETIME
    Public ftLastAccessTime As FILETIME
    '''FILETIME->_FILETIME
    Public ftLastWriteTime As FILETIME
    '''DWORD->unsigned int
    Public nFileSizeHigh As UInteger
    '''DWORD->unsigned int
    Public nFileSizeLow As UInteger
    '''DWORD->unsigned int
    Public dwReserved0 As UInteger
    '''DWORD->unsigned int
    Public dwReserved1 As UInteger
    '''WCHAR[260]
    <System.Runtime.InteropServices.MarshalAsAttribute( _
        System.Runtime.InteropServices.UnmanagedType.ByValTStr, SizeConst:=260)> _
    Public cFileName As String
    '''WCHAR[14]
    <System.Runtime.InteropServices.MarshalAsAttribute( _
        System.Runtime.InteropServices.UnmanagedType.ByValTStr, SizeConst:=14)> _
    Public cAlternateFileName As String
End Structure

<System.Runtime.InteropServices.StructLayoutAttribute( _
    System.Runtime.InteropServices.LayoutKind.Sequential)> _
Public Structure FILETIME
    '''DWORD->unsigned int
    Public dwLowDateTime As UInteger
    '''DWORD->unsigned int
    Public dwHighDateTime As UInteger
End Structure

Partial Public Class NativeMethods
    '''Return Type: HANDLE->void*
    '''lpFileName: LPCWSTR->WCHAR*
    '''lpFindFileData: LPWIN32_FIND_DATAW->_WIN32_FIND_DATAW*
    <System.Runtime.InteropServices.DllImportAttribute("kernel32.dll", EntryPoint:="FindFirstFileW")> _
    Public Shared Function FindFirstFileW( _
        <System.Runtime.InteropServices.InAttribute(), _
            System.Runtime.InteropServices.MarshalAsAttribute(System.Runtime.InteropServices.UnmanagedType.LPWStr)> _
            ByVal lpFileName As String, _
        <System.Runtime.InteropServices.OutAttribute()> _
        ByRef lpFindFileData As WIN32_FIND_DATAW) As System.IntPtr
    End Function
End Class

Translating Large Code bases

The snippet translator works well for small snippets of code. If you are trying to translate a much larger code base, say several interdependent header files the small snippet dialog won’t work well. To work with larger code bases you should use the command line version of the tool; sigimp.exe. It is designed to process several header files and produce a mass output.

Wrapping Up

This tool started out as a pet project of mine some time ago. I’m extremely excited that customers are now going to be able to take advantage of it and I greatly look forward to any feedback you have. I will post a couple more articles in the future detailing how this tool works under the hood.

Pipesinglereadernolock

2008-03-03T00:00:00+00:00

Previously we discussed a multi-thread safe queue like data structure using locks as an internal synchronization mechanism. This time we’ll look at a version requiring no locks.

In the previous version, locks were used to synchronize access to an underlying queue which stored the data. Removing the locks necessitates us moving away from Queue to store the data and forces us onto another structure which is safe in the presence of multiple threads.

Immutable collections are safe in the presence of multiple threads and don’t require any synchronization mechanisms. However they’re immutable and we’re trying to build a mutable data structure? No matter. Even though each instance is immutable they can be used to create new instances which represent a somewhat mutated state.

For this exercise we will be using a slight variant of Eric Lippert’s Immutable Stack implementation ¹. The pipe will have two stacks; 1) for capturing input and 2) for storing output. Named m_writeStack and m_readStack respectively.

This makes the implementation of reading input straight forward. While m_readStack is not empty the reader thread can systematically pop off the values. This doesn’t require any contention with writer threads and since their is only one reader thread and the stack is immutable the code is straight forward. Once the m_readStack is empty the reader thread will swap out the current state of m_writeStack with an empty stack. The original value will be reversed so we can maintain the FIFO ordering and set as the new m_readStack.

private bool CheckForInput() {
    if( m_readerStack.IsEmpty ) {
        var prev = Interlocked.Exchange(ref m_writerStack, ImmutableStack<T>.Empty);
        m_readerStack = prev.Reverse();
    }
    return !m_readerStack.IsEmpty;
}

Writing data is more complicated because it must deal with contention for updating the same data structure. It can be altered by other writers or the reader thread when it runs out of data. Basically it must push a value onto the stack and update the m_writerStack to point to the new value. In between the operations the value of m_writerStack could change and thus invalidate the push. To guard against this the writer must guarantee the current value of the m_writerStack is the same as it was before the push operation. Interlocked.CompareExchange will do the trick. If it’s been changed then repeat the operation.

public void AddInput(T value) {
    bool done;
    do {
        if (m_inputClosed) {
            throw new InvalidOperationException("Input end of pipe is closed");
        }
        var originalStack = m_writerStack;
        var newStack = originalStack.Push(value);
        var currentStack = Interlocked.CompareExchange(ref m_writerStack, newStack, originalStack);
        done = object.ReferenceEquals(currentStack, originalStack);
    } while (!done);
    m_event.Set();
}

At a glance it may seem like this suffers from the ABA problem. This is not the case and it’s easy to prove. In between the read and push operation only two other operations can modify the m_writeStack variable. The first is another write which will produce a new value of ImmutableStack. In this case the CLR guarantees that the references will not be equal and the CAS operation won't succeed. The second is the reader thread swaps out the value and replaces it with Empty. It's possible to hit an ABA situation here (detailed below) but fundamentally if it was empty before and empty now the operation is still safe. No data is lost because we are still replacing an empty stack with a single value'd stack.

Here’s a more elaborate version starting with an empty pipe of type int

Thread 1: Begins to write 5 and is stopped just before the CAS operation 1. originalStack points to ImmutableStack.Empty
Thread 2: Starts and completes a write of the value 6
Thread 3: Reads a value from the pipe, having no data replaces m_writeStack with ImmutableStack.Empty
Thread 1: Resumes and the CAS succeeds even though the m_writeStack technically changed in the middle of the operation

Even though this exhibits many characterstics of the ABA pattern it is not a problem because the behavior is still correct. No data was lost because it was transferred to the reader thread. The end result is m_writeStack pointing to a single vaue’d stack containing the data 5 which is valid. The pipe does not guarantee the ordering of data between writer threads (merely the ordering between a single writer thread).

Below is the implementation in it’s entirety.

public class PipeSingleReaderNoLock<T> : IDisposable {
    private readonly ThreadAffinity m_affinity = new ThreadAffinity();
    private ImmutableStack<T> m_readerStack = ImmutableStack<T>.Empty;
    private ImmutableStack<T> m_writerStack = ImmutableStack<T>.Empty;
    private readonly AutoResetEvent m_event = new AutoResetEvent(false);
    private volatile bool m_inputClosed;

    public PipeSingleReaderNoLock() {
    }

    #region Dispose
    public void Dispose() {
        Dispose(true);
        GC.SuppressFinalize(this);
    }
    ~PipeSingleReaderNoLock() {
        Dispose(false);
    }
    private void Dispose(bool disposing) {
        if (disposing) {
            m_event.Close();
        }
    }
    #endregion
    public void WaitForOutput() {
        m_affinity.Check();
        while (!CheckForInput()) {
            m_event.WaitOne();
        }
    }
    public T GetNextOutput() {
        m_affinity.Check();
        T data;
        while (!TryGetOutput(out data)) {
            m_event.WaitOne();
        }
        return data;
    }
    public bool TryGetOutput(out T value) {
        m_affinity.Check();
        if (CheckForInput()) {
            value = m_readerStack.Peek();
            m_readerStack = m_readerStack.Pop();
            return true;
        }
        else {
            value = default(T);
            return false;
        }
    }
    private bool CheckForInput() {
        if (m_readerStack.IsEmpty) {
            var prev = Interlocked.Exchange(ref m_writerStack, ImmutableStack<T>.Empty);
            m_readerStack = prev.Reverse();
        }
        return !m_readerStack.IsEmpty;
    }
    public void AddInput(T value) {
        bool done;
        do {
            if (m_inputClosed) {
                throw new InvalidOperationException("Input end of pipe is closed");
            }
            var originalStack = m_writerStack;
            var newStack = originalStack.Push(value);
            var currentStack = Interlocked.CompareExchange(ref m_writerStack, newStack, originalStack);
            done = object.ReferenceEquals(currentStack, originalStack);
        } while (!done);
        m_event.Set();
    }
    public void CloseInput() {
        m_inputClosed = true;
    }

If you haven’t read this series you really should. ↩

Pipesinglereader

2008-03-02T00:00:00+00:00

Before we can get to building an Active Object implementation, there are some more primitive structures we need to define. Active Objects live on a separate thread where every call is executed in a serialized fashion on that thread. The next primitive will allow us to easily pass messages in the form of delegates from the caller to the background thread.

The structure must support adding messages from N callers but only needs to support 1 reader (the active object). Successive writes from the same thread should add the items in their respective order. The input end of the pipe can be closed while leaving the output end active. This allows for the consumer to deterministically reach an end state while not ignoring any any input.

The name of the structure is PipeSingleReader.

Designing a thread safe mutable structure is significantly different from a non-thread safe or immutable data structure. There is a significant temptation to apply existing patterns. We should question every pattern we apply to these collections. Many of these patterns lead us to design API’s which encourage bad programming practices.

Our existing collection patterns are designed around structures which behave deterministically on any given thread because they are 1) Immutable or 2) designed for single thread use only. Many of these concepts do not apply to mutable thread safe collections because they are constantly being accessed and mutated by multiple threads. This gives them the _appearance _of behaving non-deterministically with respect to a given thread.

Take the member Count for instance. This is found on virtually every collection class in the BCL. Yet having it on a mutable thread safe collection class only leads to programming errors. The only reason to have a member such is Count is to use it to make a decision. However in a mutable thread-safe collection making a decision off of this value is wrong. The value can and will change between any two instructions in code.

Take the example below. Just because the Count is >0 in the if block has no dependable relevance to what the value will be inside the if block.

var ThreadSafeList<int> col = GetList();
if( col.Count > 0 )
{
    //...
}

To guard against this and have users of the collections avoid the pit of failure members such as Count should not appear on mutable thread-safe collections.

Instead we’ll design API’s around the functionality of this structure. The input end of the structure is straight forward. All writers want is to add data.

The output end is more interesting. The end goal is to read output from the pipe but how to deal with cases where there is no data. Some programs will want to check for data, others block until data is available. We can define three methods to satisfy most scenarios

WaitForOutput - void method which blocks until input is available
GetNextOutput - Blocks until input is available and returns it
TryGetOutput - Returns immediately. If output is available it will be returned.

public class PipeSingleReader<T> : IDisposable {
    private readonly ThreadAffinity m_affinity = new ThreadAffinity();
    private readonly Queue<T> m_queue = new Queue<T>();
    private readonly AutoResetEvent m_event = new AutoResetEvent(false);
    private readonly object m_lock = new object();
    private bool m_inputClosed;

    public PipeSingleReader() {
    }

    #region Dispose
    public void Dispose() {
        Dispose(true);
        GC.SuppressFinalize(this);
    }
    ~PipeSingleReader() {
        Dispose(false);
    }
    private void Dispose(bool disposing) {
        if (disposing) {
            m_event.Close();
        }
    }
    #endregion
    public void WaitForOutput() {
        m_affinity.Check();
        do {
            lock (m_lock) {
                if (m_queue.Count > 0) {
                    return;
                }
            }
            m_event.WaitOne();
        } while (true);
    }
    public T GetNextOutput() {
        m_affinity.Check();
        T data;
        while (!TryGetOutput(out data))
        {
            m_event.WaitOne();
        } 
        return data;
    }
    public bool TryGetOutput(out T value) {
        m_affinity.Check();
        lock (m_lock) {
            if (m_queue.Count == 0) {
                value = default(T);
                return false;
            }

            value = m_queue.Dequeue();
            return true;
        }
    }
    public void AddInput(T value) {
        lock (m_lock) {
            if (m_inputClosed) {
                throw new InvalidOperationException("Input end of pipe is closed");
            }
            m_queue.Enqueue(value);
        }
        m_event.Set();
    }
    public void CloseInput() {
        lock (m_lock) {
            m_inputClosed = true;
        }
    }
}

At a quick glance it may seem odd in GetNextOutput that I loop around m_event being set and TryGetOutput. Why loop? Shouldn’t a single check for the settness of m_event be enough? In this case no. The reason why is TryGetOutput will remove output from the queue without resetting the settness of m_event. Thus m_event can be set without actually having any data in m_queue. In general the implementation must treat m_event being set as the possibility of data rather than a guarantee.

This implementation uses locks to synchronize access to the data. In general I’m a fan of avoiding locks when possible since it’s very easy to miss trivial cases. Next time we’ll look at an implementation of PipeSingleReader which avoids the use of locks.

Synchronizationcontext And Higher Order Functions

2008-02-24T00:00:00+00:00

It’s often useful to ensure that actions occur on specific threads, in particular event handlers. Take Windows Forms for instance where all operations on a Control must occur on the thread it was created on. Typically this is not a problem since WinForms respond to events such as Click, Move, etc… These events are sourced from the same thread so it’s not an issue.

But there are cases where events are sourced from a separate thread and we need to Marshal it back onto the Control thread. One good example of this is FileSystemWatcher. If a SynchronizationObject is not provided it will raise the event on an unspecified thread. This event cannot directly touch a Control or an “Illegal cross thread call exception” will occur. Many examples use ISynchronizedInvoke to marshal the code back. There are a couple of downsides to this approach including

ISynchronizedInvoke on Controls won’t work until the Handle is created or after it’s destroyed. So if the event fires in either of these cases an unhandled exception will occur and typically crash the process.
Can’t use an anonymous lambda because ISynchronizedInvoke is not typed to a specific delegate
Code is easy to get subtly wrong

Here is an example implementation.

private void OnFileChanged(object sender, FileSystemEventArgs e)
{
    if( this.InvokeRequired )
    {
        // If the handle is not created this will throw
        Invoke((MethodInvoker)(() => OnFileChanged(sender, e)));
        return;
    }

    textBox2.Text = String.Format("{0} {1}", e.ChangeType, e.Name);
}

It would be easier if we could bind a delegate to a particular thread in such way that calls automatically marshal to the appropriate thread. Imagine for instance if we could type the following in such a way that all invocations of “del” below would automatically marshal to the thread for the Control. We could then freely pass this to any event source and not have to worry about what thread the event is raised on.

var del = SynchronizationContext.Current.BindDelegateAsPost(new FileSystemEventHandler(OnFileChanged));

Instead of ISynchronizedInvoke we’ll use SynchronizationContext. IMHO this is a better approach for this type of work. It has the same functionality as ISynchronizedInvoke and helps with a few of the quirks. The Windows Forms Application Model (and if memory serves WPF) insert a SynchronizationContext for every thread running a WinForm application. It greatly reduces the chance your code will run into problem #1 above because the timespan for when it can be used to Marshal between threads is not dependent upon the internal workings of a particular Control. Instead it’s tied to the lifetime of the Thread¹.

The basic strategy we’ll take is to create a new delegate which wraps the original delegate. This will Marshal the call onto the appropriate thread and then call the original delegate. SynchronizationContext has two methods to Marshal calls between threads; Post and Send.

Creating a delegate instance on the fly is not straight forward. Unless we code all permutations of delegate signatures into a class we cannot use the Delegate.Create API because we cannot provide a method with the matching signature. Instead we need to go through Reflection.Emit. This allows us to build a method on the fly to match the delegate signature. In addition we can generate the IL to route the code through Post/Send before calling the delegate.

First up are extension methods for SynchronizationContext that call into a helper class.

public static T BindDelegateAsPost<T>(this SynchronizationContext context, T del)
{
    return DelegateFactory.CreateAsPost(context, del);
} 
public static T BindDelegateAsSend<T>(this SynchronizationContext context, T del)
{
    return DelegateFactory.CreateAsSend(context, del);
}

Next is a class which injects the Send/Post call. We need this as a storage mechanism for holding the context and delegate. Essentially this is a hand generate closure.

private class DelegateData
{
    private SynchronizationContext m_context;
    private Delegate m_target;

    internal DelegateData(SynchronizationContext context, Delegate target)
    {
        m_target = target;
        m_context = context;
    }

    public void Send(object[] args)
    {
        m_context.Send(() => m_target.DynamicInvoke(args));
    }

    public void Post(object[] args)
    {
        m_context.Post(() => m_target.DynamicInvoke(args));
    }
}

Now comes the actual delegate generation. The dynamic method will be bound to an instance of the DelegateData class. As such we must add an additional parameter to the delegate of type DelegateData in position 0. The rest of the method creates an object array with length equal to the number of parameters in the delegate. Each of the arguments are added to this array. Then it will call Post/Send in DelegateData passing the arguments along.

private static T Create<T>(SynchronizationContext context, T target, string name)
{
    Delegate del = (Delegate)(object)target;
    if (del.Method.ReturnType != typeof(void))
    {
        throw new ArgumentException("Only void return types currently supported");
    }

    var paramList = new List<Type>();
    paramList.Add(typeof(DelegateData));
    paramList.AddRange(del.Method.GetParameters().Project((x) => x.ParameterType));
    var method = new DynamicMethod(
        "AMethodName",
        del.Method.ReturnType,
        paramList.ToArray(),
        typeof(DelegateData));
    var gen = method.GetILGenerator();
    var localInfo = gen.DeclareLocal(typeof(object[]));
    gen.Emit(OpCodes.Ldc_I4, paramList.Count - 1);
    gen.Emit(OpCodes.Newarr, typeof(object));
    gen.Emit(OpCodes.Stloc, localInfo.LocalIndex);
    for (int i = 1; i < paramList.Count; ++i)
    {
        gen.Emit(OpCodes.Ldloc, localInfo.LocalIndex);
        gen.Emit(OpCodes.Ldc_I4, i - 1);
        gen.Emit(OpCodes.Ldarg, i);
        if (paramList[i].IsValueType)
        {
            gen.Emit(OpCodes.Box);
        }
        gen.Emit(OpCodes.Stelem_Ref);
    }

    gen.Emit(OpCodes.Ldarg_0);
    gen.Emit(OpCodes.Ldloc, localInfo.LocalIndex);
    gen.EmitCall(OpCodes.Call, typeof(DelegateData).GetMethod(name, BindingFlags.Instance | BindingFlags.Public), null);
    gen.Emit(OpCodes.Ret);
    return (T)(object)method.CreateDelegate(typeof(T), new DelegateData(context, del));
}

internal static T CreateAsSend<T>(SynchronizationContext context, T target)
{
    return Create(context, target, "Send");
}

internal static T CreateAsPost<T>(SynchronizationContext context, T target)
{
    return Create(context, target, "Post");
}

The resulting delegate is now of the same type as the original delegate and invocations will occur on the targeted thread.

Granted if you try to use a SynchronizationContext to Marshal between threads after the target thread has finished you will still get an error. ↩

Factory Methods For Futures

2008-02-23T00:00:00+00:00

Like most generic classes, I prefer to create Future instances through static factory methods which allows me to take maximum advantage of type inference.

In addition to the 2 straight forward declaration of Func and Action, the methods will include overloads which take in Func with varying numbers of arguments. The overloads will cury the arguments and create a Func taking no arguments. This is a great convenience to the user and takes little extra code to implement.

In addition because we don’t expose the EmptyFuture class directly we need to provide a factory method to create it in a non-run state. Otherwise we are forcing the EmptyFuture to always be created and run in the ThreadPool.

public static Future Create(Action action)
{
    var f = new EmptyFuture(action);
    f.RunInThreadPool();
    return f;
}

public static Future Create<TArg1>(Action<TArg1> action, TArg1 arg1)
{
    return Create(() => action(arg1));
}

public static Future<T> Create<T>(Func<T> func)
{
    var f = new Future<T>(func);
    f.RunInThreadPool();
    return f;
}

public static Future<TReturn> Create<TArg1,TReturn>(Func<TArg1, TReturn> func, TArg1 arg1)
{
    return Create(() => func(arg1));
}

public static Future<T> CreateNoRun<T>(Func<T> func)
{
    return new Future<T>(func);
}

public static Future CreateNoRun(Action action)
{
    return new EmptyFuture(action);
}

Thread Affinity

2008-02-22T00:00:00+00:00

Part of creating a multithreading program is understanding which threads objects live on. Seems simple enough and typically is. However it’s nice to insert guarantees to match the design.

One type of threading model is for objects or subsets of methods in an object to have an affinity to a particular thread. Meaning that those methods can only be validly called on a particular thread and no other. I create a small helper object to help validate this affinity.

Simple, but gets the job done.

[Immutable]
public sealed class ThreadAffinity
{
    private readonly int m_threadId;

    public ThreadAffinity()
    {
        m_threadId = Thread.CurrentThread.ManagedThreadId;
    }

    public void Check()
    {
        if (Thread.CurrentThread.ManagedThreadId != m_threadId)
        {
            var msg = String.Format(
                "Call to class with affinity to thread {0} detected from thread {1}.",
                m_threadId,
                Thread.CurrentThread.ManagedThreadId);
            throw new InvalidOperationException(msg);
        }
    }
}

Multiple Paths To Iunknown

2008-02-22T00:00:00+00:00

ATL has a lot of great tools for COM programming and CComPtr is a good example. It’s a smart pointer class which manages the reference count of an underlying COM object.

One of it’s limitations though is it will only work properly when the inheritance chain for a class has only one path to IUnknown. If it has more than one path, the following error will be issued when you attempt to assign a value of type T to the CComPtr.

error C2594: 'argument' : ambiguous conversions from 'SomeClass *' to 'IUnknown *'  

The reason behind this is the operator= for CComPtr uses AtlComPtrAssign to change the references. The right hand side of the assignment is passed to this function as IUnknown. Since there are multiple paths to IUnknown the C++ compiler cannot implicitly perform the cast and issues the above error.

I most frequently encounter this error in larger code bases with older classes. New functionality is needed so I add a new interface and end up with a lot of C2594 errors.

To work around this I defined a new CComPtr class named CComPtrEx which inherits from CComPtr base. It defines the same operators as CComPtr but uses a Copy Constructor and Swap to perform the = which gets around the multiple paths to IUnknown. The rest of the functions are identical to CComPtr.

template <class T>
class CComPtrEx : public CComPtrBase<T>
{
public:
    CComPtrEx() throw()
    {
    }
    CComPtrEx(int nNull) throw() :
        CComPtrBase<T>(nNull)
    {
    }
    CComPtrEx(T* lp) throw() :
        CComPtrBase<T>(lp)
    {
    }
    CComPtrEx(_In_ const CComPtrEx<T>& lp) throw() :
        CComPtrBase<T>(lp.p)
    {
    }

    T* operator=(_In_opt_ T* lp) throw()
    {
        if(*this!=lp)
        {
            CComPtrEx<T> sp(lp);
            Swap(&p, &sp.p);
        }
        return *this;
    }

    T* operator=(_In_ const CComPtrEx<T>& lp) throw()
    {
        if(*this!=lp)
        {
            CComPtrEx<T> sp(lp);
            Swap(&p, &sp.p);
        }
        return *this;
    }
};

Building A Future Which Returns No Value

2008-02-18T00:00:00+00:00

In addition to [Future](/2008/02/13/building-future-t.html) there is also the concept of Futures that don't return any values. Instead the perform the operation and return. Because there is no additional data to pass between the threads building an Empty Future is fairly straight forward.

The biggest decision is how to declare it. EmptyFuture exposes no new accessible methods or properties above what Future already exposes. If a class doesn’t provide any additional behavior is there any reason to expose it’ In this case I think not. Therefore it will be declared as a private inner class of Future.

Yes this makes it impossible to directly create from a user perspective. You could also make a good argument that creation is new behavior and therefore EmptyFuture should be exposed. However for Future’s, as with other generic classes, I prefer static factory methods for creation. It allows the user to take advantage of type inference as much as possible.

private sealed class EmptyFuture : Future
{
    private Action m_action;

    internal EmptyFuture(Action action)
    {
        m_action = action;
    }

    protected override void RunCore()
    {
        m_action();
    }

}

Next we’ll go over the creation of the factory methods to create this and Future.

Building Future T

2008-02-13T00:00:00+00:00

The last post dealt with building the base Future class. Now we’ll build the child class used to run [Func](http://msdn2.microsoft.com /en-us/library/bb534960.aspx)'s.

The basic implementation is straight forward. The class will run a delegate typed to [Func](http://msdn2.microsoft.com/en-us/library/bb534960.aspx) in the override of RunCore. The trickiest part is how to store the value. The value is set on one thread and read off of another.

When a value is read and written on multiple threads there are a couple of options for synchronization between threads. One of them is to use the volatile keyword for the data. This forces the CLR to read the value from memory every time and prevents caching issues between threads. Unfortunately volatile cannot be applied to an unbounded generic.

To get around this I’ve declared the value to be of type object. Whenever the value is accessed by the user of Future a cast is applied to the appropriate type. This incurs boxing overhead but it's minimal and in the typical case will be limited to one box and unbox per value type.

In addition Future adds one new method; Wait;?? It's a combination of calling WaitEmpty followed by returning the value.

In a perfect world WaitEmpty in Future would really be called Wait and be virtual. Future would override the method and alter the return type to be T. Unfortunately C#/VB don't support covariant return types on virtual method overrides so it's not possible. Truthfully I don't know if this is a C#/VB limitation or a CLR one.

public class Future<T> : Future
{
    private Func<T> m_function;
    private volatile object m_value;

    public T Value
    {
        get { return Wait(); }
    }

    public Future(Func<T> function)
    {
        m_function = function;
    }

    public T Wait()
    {
        base.WaitEmpty();
        return (T)m_value;
    }

    protected override void RunCore()
    {
        m_value = m_function();
    }
}

Next time I’ll go over the implementation of Futures which return no values.

Building The Base Future

2008-02-12T00:00:00+00:00

In the end there are two basic types of Future implementations you can use.

Futures which return no values
Futures which return a value

The rest of the behavior and shape of the Future is the same and screams for a pattern of sorts. I’ve found the best way to implement this behavior is through an inheritance pattern. The base class is name of course Future. It’s purpose is to provide a common way in which to schedule the invocation of a delegate, handle exceptions, wait for the completion of such delegate and enforce certain contracts such as not running the future more than once.

At the core it only needs a few members.

ActiveOperation m_operation which is used to implement the waiting portion
int m_run used to ensure a future is not run twice
Exception m_error to record any exceptions thrown by running the delegate

private ActiveOperation m_operation = new ActiveOperation();
private int m_run;
private Exception m_error;

It has one public property to determine whether or not a Future has completed. It’s a proxy into m_operation

public bool HasCompleted
{
    get { return m_operation.HasCompleted; }
}

By using m_operation to deal with Waiting, the majority of WaitEmpty can be proxied to m_operation as well. The only additional work needed is to deal with Exceptions.

public void WaitEmpty()
{
    m_operation.Wait();
    if (m_error != null)
    {
        throw new FutureException("Error occurred running future", m_error);
    }
}

The only behavior a child class needs is a place to invoke the delegate. A single abstract method is provided to allow implement this behavior.

protected abstract void RunCore();

Before calling this method the base Future class must make sure that all of the contracts are met. This is implemented through a wrapper method around RunCore. It is the only method that calls RunCore directly.

private void RunWrapper()
{
    if (0 != Interlocked.CompareExchange(ref m_run, 1, 0))
    {
        throw new InvalidOperationException("Future is already run");
    }

    try
    {
        RunCore();
    }
    catch (Exception ex)
    {
        Interlocked.Exchange(ref m_error, ex);
    }
    finally
    {
        m_operation.Completed();
    }
}

It’s very important that m_operation is signalled as completed after exception handling occurs. The setting or not setting of m_error is the only way we know if an exception occurred when waiting. If we do this in the opposite order it’s possible for WaitEmpty() to complete before the exception is set and hence miss the error.

Lastly is the code to actually run the Future. There are two ways to run the Future (more can be added).

Asynchronously through the ThreadPool. This is the more common case.
Synchronously on the same thread. This will mainly be used to implement such operations as methods in Active Objects.

public void Run()
{
    RunWrapper();
}

public void RunInThreadPool()
{
    ThreadPool.QueueUserWorkItem((x) => RunWrapper());
}

This leaves us with a base class implementation for Future’s. Next we’ll implement Future which return values.

Dealing With Exceptions In A Future

2008-02-11T00:00:00+00:00

Besides waiting, the another important issue when dealing with Futures is how to deal with exceptions thrown by the user specified code.

Option 1: Ignore the Exception

Don’t take any actions in the future code and force users to write exception free code. IMHO this is not the best way to approach the problem. The code will be running in the thread pool and unhandled exceptions in the thread pool result in the taking down of an appdomain/process. In addition Futures are designed to be simple. Adding a try/catch around every lambda is not practical and/or readable.

Option 2: Catch and Swallow

Catch the exception on the background thread and swallow it. Silently failing is in many cases worse than actually crashing. Behavior will become flaky and the user/developer won’t have any indication there is an error.

Option 3: Re-throw the Exception when Wait is called

Catch and save the exception when it occurs on the background thread. Then when Wait() is called on a Future re-throw the exception. This makes exception handled deterministic.

It’s also very similar to the exception handling semantics of calling a method. The only difference is that users must handle the exception at the point of method completion vs invocation. For synchronous methods this is just the same point.

The big downside to this approach is the stack trace information is lost from the exception. Re-throwing will instead add the stack trace at the point of the re-throw. Not having stack trace information makes it very difficult to actually track down the source of an error.

**Option 4: Re-throw a new Exception when Wait is called **

This is very similar to Option #3. The only difference is when the user calls Wait, throw a new exception and make the original exception an inner exception of the new one. We’ll call this exception FutureException. This has the advantages of option 3 and in addition will preserve the stack trace information from the original exception.

There is a downside to this approach though. Users can no longer have different catch blocks to handle the different types of exceptions that can be thrown.

try
{
    Future.Create(() => SomeOperation());
}
catch (IOException ex)
{
    // ...
}
catch (InvalidOperationException ex)
{
    // ...
}

Instead the user can only catch a Future exception and examine the inner result to take corrective action.

try
{
    Future.Create(() => SomeOperation());
}
catch (FutureException ex)
{
    var type = ex.InnerException.GetType();
    if (type == typeof(IOException))
    {
        // ...
    }
    else if (type == typeof(InvalidOperationException))
    {
        // ...
    }
}

This doesn’t actually limit any functionality but users may find the syntax uncomfortable. VB users can still do exception filtering but this is not at option for C# users.

Try
    Future.Create(Function() SomeOperation())
Catch ex As Exception When ex.InnerException.GetType() Is GetType(IOException)

End Try

The FutureException class is straight forward. A simple implementation of the exception snippet will do the trick.

[global::System.Serializable]
public class FutureException : Exception
{
    public FutureException() { }
    public FutureException(string message) : base(message) { }
    public FutureException(string message, Exception inner) : base(message, inner) { }
    protected FutureException(
      System.Runtime.Serialization.SerializationInfo info,
      System.Runtime.Serialization.StreamingContext context)
        : base(info, context) { }
}

Push Enumerators

2008-02-05T00:00:00+00:00

If you read Jon Skeet’s blog you’ll notice he’s been playing around lately with “push” style enumerators. Push enumerators are the concept of “we’ll tell you when we’re ready”. This is different from IEnumerator which is more of a pull; "ask me if I have more data model".

His latest idea is an interface based approach called IDataProducer. The full post can be found here. <http://msmvps.com/blogs/jon.skeet/archive/2007/11/29/group-pipelining-returns-new-and-improved-design.aspx>.

After reading the post I decided to try and bridge the gap between the worlds of Push/Pull models by defining an enumerator over an IDataProducer. Integration with the existing paradigms is extremely important for the adoption of a new technology. Eventually you'll eventually want to pass your asynchronous enumerator off to an API requiring IEnumerable.

Since we are implementing a pull model it will revolve around asking for more data and blocking until such data is available. To implement the wait functionality I’ve used an AutoResetEvent to provide signaling between threads. One of the tricky aspects is that IDataProducer can live on any thread and hence any thread can be raising the various events.

Below is my first attempt at getting it to work. It has a couple of defects.

I don’t dispose of the AutoResetEvent. Reason being that it’s possible for a consumer of IEnumerable to only consume part of the data. If they only consume half of the data and then dispose of the EnumerableDataProducer and another event comes I'll be accessing a disposed WaitHandle. You can work around this but I wanted to keep it simpler for now.
It’s not Reset-able. The generated IEnumerator throws a NotSupportedException anyway but we'd have to do a bit of work to make this resetable.

public class EnumerableDataProducer<T> : IEnumerable<T>
{
    private object m_lock = new object();
    private bool m_finished;
    private AutoResetEvent m_event = new AutoResetEvent(false);
    private Queue<T> m_queue = new Queue<T>();

    public EnumerableDataProducer(IDataProducer<T> producer)
    {
        producer.DataProduced += new Action<T>(OnDataProdecuded);
        producer.EndOfData += new Action(OnEndOfData);
    }

    private void OnDataProdecuded(T obj)
    {
        lock (m_lock)
        {
            m_queue.Enqueue(obj);
        }

        m_event.Set();
    }

    private void OnEndOfData()
    {
        lock (m_lock)
        {
            m_finished = true;
        }
        m_event.Set();
    }

    #region IEnumerable<T> Members

    public IEnumerator<T> GetEnumerator()
    {
        while (true)
        {
            bool needWait = false;
            lock (m_lock)
            {
                if (m_finished)
                {
                    break;
                }

                if (m_queue.Count > 0)
                {
                    yield return m_queue.Dequeue();
                }
                else
                {
                    needWait = true;
                }
            }

            if (needWait)
            {
                m_event.WaitOne();
            }
        }
    }

    #endregion

    #region IEnumerable Members

    System.Collections.IEnumerator System.Collections.IEnumerable.GetEnumerator()
    {
        return GetEnumerator();
    }

    #endregion
}

The First Part Of Building A Future Is Waiting

2008-02-04T00:00:00+00:00

Future’s are a great abstraction for asynchronous programming. One of the items making them so good is the easy manner in which you can declare one and wait for it to finish. The idea is to allow for many futures to be declared with as little overhead as possible. In order to do so you need to define an efficient way of waiting.

Normally when you need to wait on operations between two threads to complete you use a Thread.Join call or a form of a WaitHandle. I tend to prefer a ManualResetEvent.

Unfortunately a ManualResetEvent is not free and under the hood will allocate a kernel handle. There are a lot of handles to go around and while unlikely that a program purely using futures will allocate too many handles, you may be running with other code that is handle hungry.

In addition several cases do not need a handle. For instance in the below code, if the commented out section takes longer than the future to complete then why do you even need wait handle of any sort?

var d = Future.Create(() => CallASimpleFunction);
// ...
d.Wait();

Therefore the first step is to define an efficient waiting mechanism. I call it an ActiveOperation. It provides 3 basic methods; HasCompleted, Completed and Wait. It optimizes for trying to not created a WaitEvent unless actually necessary. It has two member variables. An int for completion check and a ManualResetEvent to be used for shared waiting when necessary. Notice that m_hasCompleted is not volatile, instead all writes use a Interlocked operation to ensure it is propagated between threads.

private int m_hasCompleted;
private ManualResetEvent m_waitEvent;

HasCompleted is straightforward.

public bool HasCompleted
{
    get { return m_hasCompleted == 1; }
}

Completed is a little bit trickier. It has to deal with a couple of cases.

Another thread already called Completed.
Completed called before another thread calls Wait.
Completed called while or after another thread calls Wait.

public void Completed()
{
    if (0 == Interlocked.CompareExchange(ref m_hasCompleted, 1, 0))
    {
        ManualResetEvent mre = m_waitEvent;
        if (mre != null)
        {
            try
            {
                mre.Set();
            }
            catch (ObjectDisposedException)
            {
                // If another thread is in Wait at the same time and sees the completed flag
                // it may dispose of the shared event.  In this case there is no need to signal
                // just return.
            }
        }
    }
}

Wait is the trickiest one. It has the following cases to deal with

HasCompleted already set
Second or later thread to call Wait and needs to wait on m_waitEvent
While attempting to create m_waitEvent, another thread in Wait finishes first.
Thread successfully creates and owns the shared m_waitEvent variable before Completed() is called.
During the creation of m_waitEvent, another thread calls Completed() in which case there is no guarantee that m_waitEvent will be signaled.

public void Wait()
{
    // Case 1
    if (HasCompleted)
    {
        return;
    }

    // Case 2
    ManualResetEvent sharedEvent = m_waitEvent;
    if (sharedEvent != null)
    {
        WaitOnEvent(sharedEvent);
    }

    ManualResetEvent created = null;
    try
    {
        created = new ManualResetEvent(false);
        sharedEvent = Interlocked.CompareExchange(ref m_waitEvent, created, null);
        if (null != sharedEvent)
        {
            // Case 3.  Another thread got here first and it's created is now the shared event.  Wait
            // on that event
            WaitOnEvent(sharedEvent);
        }
        else if (HasCompleted)
        {
            // Case 5. In between the time we checked for completion and created the event a completion
            // occurred.  Returning will dispose of m_waitEvent and force other threads Wait to complete 
            return;
        }
        else
        {
            // Case 4. 
            WaitOnEvent(created);
        }
    }
    finally
    {
        if (created != null )
        {
            if ( sharedEvent == null)
            {
                Interlocked.Exchange(ref m_waitEvent, null);
            }

            created.Close();
        }
    }
}

private void WaitOnEvent(ManualResetEvent mre)
{
    try
    {
        mre.WaitOne();
    }
    catch (ObjectDisposedException)
    {

    }
}

Now you have one of the basic building blocks of Futures.

Get Content And File Names

2008-02-02T00:00:00+00:00

Another day, another PowerShell feature discovered. Unfortunately this time it was a feature that made me think I had a bug in my script. The script read through some directories, did some file parsing and created a data object for every directory in the form of a Tuple.

One of the files was People.txt and contained a list of relevant people for the data (one per line). Unfortunately the parameter kept showing up like so …

People  
\------  
{People.txt, People.txt, People.txt, People.txt}

Confusion set in as to what I did wrong since the code in question amounted to the following.

D:\temp> new-tuple "People",(gc People.txt)

Next step is manual verification

D:\temp> gc People.txt  
jared  
jamie  
jason  
mary

So far so good. Then a tried the explicit tuple code and got back the People.txt problem. Eventually I decided to try out get-member and see exactly what I had in the array. It was a string type as expected but then I noticed the following … extra items.

D:\temp> gc People.txt | gm

'? TypeName: System.String

Name'''''''' MemberType'''''''? Definition  
\----'''''''' \----------'''''''? \----------  
...

PSParentPath''?? NoteProperty'''''' System.String PSParentPath=D:\temp  
PSPath''''''?? NoteProperty'''''' System.String PSPath=D:\temp\People.txt  
PSProvider'''' NoteProperty
System.Management.Automation.Provider...  
ReadCount''''?? NoteProperty'''''' System.Int64 ReadCount=1  

It turns out that when you read information from a file in PowerShell they will handily add in the file information as NoteProperty instances. All well and good and I can see where that would be useful but for now it’s really messing up my display. To remove it just tell powershell that I really just want a string.

D:\temp> new-tuple "People",(gc People.txt | %{[string]$_})

People  
\------  
{jared, jamie, jason, mary}

Active Objects And Futures

2008-01-28T00:00:00+00:00

Herb Sutter gave one of my favorite and inspiring presentations. It is called “The Free Lunch is Over”. The original article can be found here. My first encounter though came from his PDC presentation and highly recommend viewing that as well.

The part that interested me the most about the talk was two new threading abstractions I hadn’t encountered before. Future’s and ActiveObjects. One of the basic premise is that concurrency should be grep`able and somewhat declarative. The act of calling a method on a background thread and later waiting for it to complete should be simple, not complicated.

Asynchronous programming is one of my favorite aspects of computing. What interests me the most is how asynchronous programming can be useful for UI. My biggest pet peeve is when UI hangs because an operation call, or network operation takes too long. Why not make multi-threading easy and give users a way to cancel out of these operations’?? Or start loading on the background thread instead of waiting for the user to perform a specific. Hopefully over the next month or so I’ll lay out some utilities and classes building on Futures and Active Objects that will do precisely this.

Future’s are actions where work can be done now, but the result is not needed until a future time. Work occurs on a separate thread and the results can be easily joined once work is complete.

var f = Future.Create(() => LongCalculation());
// ...
var result = f.Wait();

Future’s are now exposed via the Parallel Extension team’s work. You can download the CTP off of their web site and get to work.

ActiveObjects are objects which only expose Asynchronous functions where the return value is exposed as a Future. So instead of

string GetName()

You would have

Future<string> GetName()

An ActiveObject essentially lives on or owns a thread. All operations are queued up and processed one at a time. Since only one action at a time can be executing the object internals don’t have to use locks or consider many types of race conditions. In fact if your return types are immutable a great many threading concerns go out the window. Yet all of the calls are inherently asynchronous so callers can get the result only when they are needed. The best of both worlds.

Both of these provide significant advantages over the “lock before use” patterns. In my experience I find these to be hard to maintain and lead to difficult to track down bugs. I can’t tell you how many times I’ve gone through someone else’s code, or even my own, and wondered …

Did they forget to lock here or is this an optimization?
Is a join needed here or can these terminate at separate times?
OK I need to touch that variable, can it be accessed in multiple threads or is it safe?

Futures/Active Objects on the other hand are a bit more declarative and straight forward to understand than plain old locks. They allow you to do away with many uses of plain old locking. Don’t confuse this with me saying they are a cure all for threading. They’re not. But in my experiences I’ve found them to be a significant upgrade.

Over the next month or so I’ll be laying out the design for a basic ActiveObject implementation. We will likely have to deviate off of the Parrallel Extension work to get certain behaviors but the concepts map well.

Tuples Part 8 Finishing Up

2008-01-27T00:00:00+00:00

There are only a few missing features from our tuple implementation. Mainly FxCop compliance, debugging support and test case code. The actual functional work is complete.

The one issue with FxCop compliance is the chosen names. Namely using A,B etc. FxCop, rightly, believes names should have more value. Accordingly, calling the generic argument corresponding to A, TA also causes the same issue. This is a design decision made from the begining. I don’t believe changing the name to ValueA adds any more value than simply A. Therefore the warning for this will simply be suppressed.

Additionally FxCop doesn’t like types with more than 3 generic parameters. This is also a design decision intentionally done and there is no avoiding it. It will be suppressed as well.

For debugging support a simple DebuggerDisplay attribute will be used. It will display the current value of all of the tuple values.

Here is the latest version of the full script which includes all of the new information. With the exception of a few small tweaks this is just the combination of the individual parts specified throughout these postings.

param ( [int]$tupleCount = 5, 
        [string]$namespace = "Tuples" )

$script:scriptPath = split-path -parent $MyInvocation.MyCommand.Definition 
$script:lowerList = 0..25 | %{ [char]([int][char]'a'+$_) } 
$script:upperList = 0..25 | %{ [char]([int][char]'A'+$_) } 
$script:valueList = "1","42",'"bar"', '"foo"', 'true'

function script:Gen-FxCop 
{ 
    param ( [int]$code )

    switch ( $code ) 
    { 
        1704 { '[SuppressMessage("Microsoft.Naming", "CA1704")]'; break } 
        1005 { '[SuppressMessage("Microsoft.Design", "CA1005")]'; break } 
        default: {$(throw "Invalid")} 
    } 
}

function script:Gen-Display 
{ 
    param ( [int]$count ) 
    $OFS = ", " 
    $p = [string](0..($count-1) | %{ "{0}={{{0}}}" -f $upperList[$_] }) 
    '[DebuggerDisplay("{0}")]' -f $p 
}

function script:Gen-Property 
{ 
    param ( [int] $index  = $(throw "Need an index"), [bool]$mutable = $false)

    if (-not $mutable ) 
    { 
@" 
    private readonly T{0} m_{1}; 
    $(Gen-FxCop 1704) 
    public T{0} {0} {{ get {{ return m_{1}; }} }}

"@ -f $upperList[$index],$lowerList[$index] 
    } 
    else 
    { 
@" 
    private T{0} m_{1}; 
    $(Gen-FxCop 1704) 
    public T{0} {0} {{ get {{ return m_{1}; }} set {{ m_{1} = value; }} }}

"@ -f $upperList[$index],$lowerList[$index] 
    } 
}

function script:Gen-Constructor 
{ 
    param ( [int] $count = $(throw "Need a count"), [string]$className ) 
    $OFS = ',' 
    $list = [string](0..$($count-1) | %{ "T{0} value{0}" -f $upperList[$_]}) 
    "public $className($list) {" 
    0..($count-1) | %{ "m_{0} = value{1};" -f $lowerList[$_],$upperList[$_] } 
    "}" 
}

function script:Gen-InferenceConstructor 
{ 
    param ( [int] $count = $(throw "Need a count"), [string]$name ) 
    $OFS = ',' 
    $gen = "<" + [string](0..($count-1) | %{ "T"+$upperList[$_] }) + ">"     
    $list = [string](0..$($count-1) | %{ "T{0} value{0}" -f $upperList[$_] }) 
    $argList = [string](0..$($count-1) | %{ "value{0}" -f $upperList[$_] }) 
    "public static partial class $name {" 
    Gen-FxCop 1704 
    "public static $name$gen Create$gen($list) { return new $name$gen($argList); } " 
    "}" 
}

function script:Gen-Equals 
{ 
    param ( [int] $count = $(throw "Need a count"), [string]$name ) 
    $OFS = ',' 
    $gen = "<" + [string](0..($count-1) | %{ "T"+$upperList[$_] }) + ">"     
    "public override bool Equals(object obj) { " 
    "return Equals(obj as $name$gen); }" 
    "public bool Equals($name$gen other) {" 
    "if ( Object.ReferenceEquals(other,null) ) { return false; }" 
    "if (" 
    $OFS = "&&" 
    [string](0..($count-1) | %{"EqualityComparer<T{0}>.Default.Equals(m_{1},other.m_{1})" -f $upperList[$_],$lowerList[$_] }) 
    ") { return true; }" 
    "return false;" 
    "}" 
}

function script:Gen-GetHashCode 
{ 
    param ( [int] $count = $(throw "Need a count") ) 
    "public override int GetHashCode() {" 
    "int code = 0;" 
    0..($count-1) | %{ "code += EqualityComparer<T{0}>.Default.GetHashCode(m_{1});" -f $upperList[$_],$lowerList[$_] } 
    "return code;" 
    "}" 
}

function script:Gen-ITuple 
{ 
    param ( [int] $count = $(throw "Need a count") ) 
    $OFS = ',' 
    $gen = "<" + [string](0..($count-1) | %{ "T"+$upperList[$_] }) + ">"     
    $base = "" 
    if ( $count -ne 1 ) 
    { 
        $baseGen = "<" + [string](0..($count-2) | %{ "T"+$upperList[$_] }) + ">"     
        $base = ": ITuple$baseGen" 
    } 
    else 
    { 
        $base = ": ITuple" 
    } 
    Gen-FxCop 1704 
    if ( $count -gt 2 ) { Gen-FxCop 1005 } 
    "public interface ITuple$gen $base {" 
    Gen-FxCop 1704 
    "T{0} {0} {{ get; }}" -f $upperList[$count-1]  
    "}" 
}

function script:Gen-TupleAccess 
{ 
    param ( [int] $count = $(throw "Need a count"), [bool]$mutable ) 
    "public int Count { get { return $count; } }" 
    "public object this[int index] { get { switch (index){ " 
    0..($count-1) | %{ "case $($_): return m_$($lowerList[$_]);" } 
    "default: throw new InvalidOperationException(""Bad Index"");" 
    "} }"

    if ( $mutable ) 
    { 
        "set { switch (index) {" 
        0..($count-1) | %{ "case $($_): m_$($lowerList[$_]) = (T$($upperList[$_]))value; break;" } 
        "default: throw new InvalidOperationException(""Bad Index"");" 
        "} } " 
    } 
    "}" 
}

function script:Gen-OpEquals 
{ 
    param ( [int] $count = $(throw "Need a count"), [string]$name ) 
    $OFS = ',' 
    $gen = "<" + [string](0..($count-1) | %{ "T"+$upperList[$_] }) + ">"     
    "public static bool operator==($name$gen left, $name$gen right) {" 
    "return EqualityComparer<$name$gen>.Default.Equals(left,right); }" 
    "public static bool operator!=($name$gen left, $name$gen right) {" 
    "return !EqualityComparer<$name$gen>.Default.Equals(left,right); }" 
}

function script:Gen-CompareTo 
{ 
    param ( [int] $count = $(throw "Need a count"), [string]$name ) 
    $OFS = ',' 
    $gen = "<" + [string](0..($count-1) | %{ "T"+$upperList[$_] }) + ">"     
    "public int CompareTo(object obj) {" 
    "return CompareTo(obj as $name$gen); }" 
    "public int CompareTo($name$gen other) {" 
    "if ( Object.ReferenceEquals(other,null) ) { return 1; }" 
    "int code;" 
    0..($count-1) | 
        %{ "code = Comparer<T{0}>.Default.Compare(m_{1},other.m_{1}); if (code != 0) {{ return code; }}" -f $upperList[$_],$lowerList[$_] } 
    "return 0; }" 
    "public static bool operator>($name$gen left, $name$gen right) { " 
    "return Comparer<$name$gen>.Default.Compare(left,right) > 0; }" 
    "public static bool operator<($name$gen left, $name$gen right) { " 
    "return Comparer<$name$gen>.Default.Compare(left,right) < 0; }" 
}

function script:Get-Tuple 
{ 
    param ( [int] $count, [bool]$mutable = $false ) 
    $OFS = ',' 
    $gen = "<" + [string](0..($count-1) | %{ "T"+$upperList[$_] }) + ">"     
    Gen-FxCop 1704 
    Gen-Display $count 
    if ( $count -gt 2 ) { Gen-FxCop 1005 } 
    $name = "{0}Tuple" -f (Get-Ternary $mutable "Mutable" "") 
    "public sealed class $name$gen : ITuple$gen,IEquatable<$name$gen>, IComparable<$name$gen>,IComparable {" 
    (0..($count-1) | %{ Gen-Property $_ $mutable}) 
    Gen-Constructor $count $name 
    Gen-TupleAccess $count $mutable 
    Gen-Equals $count $name 
    Gen-GetHashCode $count 
    Gen-OpEquals $count $name 
    Gen-CompareTo $count $name 
    "}" 
    Gen-InferenceConstructor $count $name 
}

function Gen-TestTuple([int]$count, [string]$prefix) 
{ 
    $OFS = "," 
    $v = [string](0..($count-1) | %{ $valueList[$_]} ) 
    "[TestMethod]" 
    "public void {0}Access{1}() {{" -f $prefix,$count 
    "var t = $prefix.Create($v);" 
    for ( $i = 0; $i -lt $count; ++$i ) 
    { 
        "Assert.AreEqual({0},t.{1});" -f $valueList[$i],$upperList[$i]; 
    } 
    "}"

    "[TestMethod]" 
    "public void {0}GenericAccess{1}() {{" -f $prefix,$count 
    "var t = $prefix.Create($v);" 
    for ( $i = 0; $i -lt $count; ++$i ) 
    { 
        ("Assert.AreEqual({0},t[{1}]);" -f $valueList[$i],$i); 
    } 
    "Assert.AreEqual({0},t.Count);" -f $i 
    "}"

    "[TestMethod]" 
    "public void {0}Equals{1}() {{" -f $prefix,$count 
    "var t1 = $prefix.Create($v);" 
    "var t2 = $prefix.Create($v);" 
    "Assert.IsTrue(t1.Equals(t2));" 
    "}"

    "[TestMethod]" 
    "public void {0}NotEquals{1}() {{" -f $prefix,$count 
    for ( $i = 0; $i -lt $count; ++$i ) 
    { 
        $leftName = "t{0}_1" -f $i 
        $rightName = "t{0}_2" -f $i 
        $left = "var $leftName = $prefix.Create(" -f $i 
        $right = "var $rightName = $prefix.Create(" -f $i 
        for ( $v = 0; $v -lt $count; ++$v ) 
        { 
            $left += "$v" 
            if ( $v -eq $i ) 
            { 
                $right += "-1" 
            } 
            else 
            { 
                $right += "$v" 
            }

            if ( ($v + 1) -lt $count ) 
            { 
                $left += "," 
                $right += "," 
            } 
        } 
        "$left);" 
        "$right);" 
        "Assert.AreEqual($leftName.GetType(), $rightName.GetType());" 
        "Assert.IsFalse($leftName.Equals($rightName));" 
    } 
    "}"

    "[TestMethod]" 
    "public void {0}CompareTest() {{ " -f $prefix 
    "Assert.IsTrue($prefix.Create(1) < $prefix.Create(2));" 
    "Assert.IsTrue($prefix.Create(2) > $prefix.Create(1));" 
    "}" 
}

$output = 
@" 
using System; 
using System.Collections.Generic; 
using System.Diagnostics; 
using System.Diagnostics.CodeAnalysis;

namespace $namespace { 
public interface ITuple { 
    int Count { get; } 
    object this[int index] { get; } 
} 
"@ 
$OFS = [Environment]::NewLine 
$output += [string](0..($tupleCount-1) | %{ Gen-ITuple ($_+1) }) 
$output += [string](0..($tupleCount-1) | %{ Get-Tuple ($_+1) }) 
$output += [string](0..($tupleCount-1) | %{ Get-Tuple ($_+1) $true }) 
$output += "}"

$output > (join-path $scriptPath "Core\Tuple.cs")

$output = 
@" 
using System; 
using System.Collections.Generic; 
using Microsoft.VisualStudio.TestTools.UnitTesting; 
using $namespace;

namespace $($nameSpace)Test {

[TestClass] 
public class TupleTest{ 
"@ 
$output += Gen-TestTuple $tupleCount "Tuple" 
$output += Gen-TestTuple $tupleCount "MutableTuple" 
$output += "}}" 
$output > (join-path $scriptPath "TestCore\TupleTest.cs")

Tuples Part 7 Mutable Tuples

2008-01-23T00:00:00+00:00

Part 6 left us with comparable tuples. At this point, the Tuple class is functionally complete. There will be a little more done with the debugability and overall fit into larger projects. But otherwise it is sound.

Now the focus shifts to generating mutable tuples. Immutability is nice for threading, memoization, etc … However it’s not always practical to use immutable objects. Often an algorithm does not benefit from immutability and lends itself to a more mutable type.

Mutable tuples behave like a Tuple in every other way except that they’re … mutable. This includes implementing interfaces as well as inheritance structure.

public sealed class MutableTuple<TA, TB> : 
    ITuple<TA, TB>, 
    IEquatable<MutableTuple<TA, TB>>, 
    IComparable<MutableTuple<TA, TB>>, 
    IComparable
{

As such the script already used will be sufficient to generate mutable classes in addition to the ones its already doing. The majority of the code difference is just in the naming of the classes. The only functional differences exist in the properties and indexer. Both of these add a setter method. Below is the modified code for generating the property and indexer.

function script:Gen-TupleAccess 
{ 
    param ( [int] $count = $(throw "Need a count"), [bool]$mutable ) 
    "public int Count { get { return $count; } }" 
    "public object this[int index] { get { switch (index){ " 
    0..($count-1) | %{ "case $($_): return m_$($lowerList[$_]);" } 
    "default: throw new InvalidOperationException(""Bad Index"");" 
    "} }"

    if ( $mutable ) 
    { 
        "set { switch (index) {" 
        0..($count-1) | %{ "case $($_): m_$($lowerList[$_]) = (T$($upperList[$_]))value; break;" } 
        "default: throw new InvalidOperationException(""Bad Index"");" 
        "} } " 
    } 
    "}" 
}


function script:Gen-Property 
{ 
    param ( [int] $index  = $(throw "Need an index"), [bool]$mutable = $false)

    if (-not $mutable ) 
    { 
@" 
    private readonly T{0} m_{1}; 
    public T{0} {0} {{ get {{ return m_{1}; }} }}

"@ -f $upperList[$index],$lowerList[$index] 
    } 
    else 
    { 
@" 
    private T{0} m_{1}; 
    public T{0} {0} {{ get {{ return m_{1}; }} set {{ m_{1} = value; }} }}

"@ -f $upperList[$index],$lowerList[$index] 
    } 
}

Now creating a mutable tuple is the same as the immutable tuple with just a name tweak.

var t1 = MutableTuple.Create("foo", 42);
t1.A = "again";

Tuples Part 6 Comparing

2008-01-22T00:00:00+00:00

Part 5 produced equality tests for Tuples. This section will add comparison support through the [IComparable](http://msdn2.microsoft.com/en-us/library/4d7sx9hd.aspx) interface. Implementing comparable is very similar to adding equality support. Once again there is a generic class available to make all of the comparison decisions for us; [Comparer](http://msdn2.microsoft.com/en-us/library/cfttsh47.aspx).

The implementation will compare objects in a left to right fashion. In this case the property corresponding to TA will be the left most, and TN the right most. If all properties are equal (Compare returns 0) then the two items will be determined to be equal and will return 0.

    function script:Gen-CompareTo  
    {  
        param ( [int] $count = $(throw "Need a count") )   
        $OFS = ','   
        $gen = "<" + [string](0..($count-1) | %{ "T"+$upperList[$_] }) + ">"       
        "public int CompareTo(object other) {"   
        "return CompareTo(other as Tuple$gen); }"   
        "public int CompareTo(Tuple$gen other) {"   
        "if ( Object.ReferenceEquals(other,null) ) { return 1; }"   
        "int code;"   
        0..($count-1) |   
            %{ "code = Comparer<T{0}>.Default.Compare(m_{1},other.m_{1}); if (code != 0) {{ return code; }}" -f $upperList[$_],$lowerList[$_] }   
        "return 0; }"   
    }

Once again the same questions arise about implementing IComparable vs IComparable (or both). The arguments are fairly similar and as a result I decided to skip implementing the ITuple version for now.

Tuples Part 5 Equality

2008-01-22T00:00:00+00:00

Part 4 left us with a reusable, abstract and inference friendly Tuple class. The next step is to be able to test for Tuple equality.

For the Tuple implementation, two tuples will be defined as equal if all of their members are equal. Seems fairly straight forward. The trick is in the implementation. In addition to doing the typical override of Equals/GetHashCode the Tuple implementation will be implementing [IEquatable](http://msdn2.microsoft.com/en-us/library/ms131187.aspx) and overloading the standard equality operators. Tuple members are all unconstrained generic classes which leaves us with a non-great starting point.

For instance what if we are dealing with value types’ Is Equals() the best method to call’ What if the type in question implements [IEquatable](http://msdn2.microsoft.com/en-us/library/ms131187.aspx) or has a well known [IEqualityComparer](http://msdn2.microsoft.com/en-us/library/ms132151.aspx)'?? What if one or both of the arguments are reference types and null? What if they're value types and equal to null?

Luckily there is an easy and straight forward solution. The BCL defines a class, [EqualityComparer](http://msdn2.microsoft.com/en-us/library/ms132123.aspx), which will properly perform equality comparisons for objects of a particular type. This makes the Equals override very straight forward.

There is one small trick to implementing Equals correctly. The implementation explicitly uses Object.ReferenceEquals to check for null rather than ==. The reason being is once operator== is defined for the type Tuple, comparison for even null will bind to this operator. Part of checking for operator== will end up calling Equals and hence you can end in a stack overflow fairly quick. Note that our implementation of == will work around this but it’s still safer to be explicit.

function script:Gen-Equals  
{  
   param ( [int] $count = $(throw "Need a count") )  
   $OFS = ','  
   $gen = "<" \+ [string](0..($count-1) | %{ "T"+$upperList[$_] }) + ">"  
   "public override bool Equals(object obj) { "  
   "return Equals(obj as Tuple$gen); }"  
   "public bool Equals(Tuple$gen other) {"  
   "if ( Object.ReferenceEquals(other,null) ) { return false; }"  
   "if ("  
   $OFS = "&&"  
   [string](0..($count-1) |
%{"EqualityComparer<T{0}>.Default.Equals(m_{1},other.m_{1})" -f
$upperList[$_],$lowerList[$_] })  
   ") { return true; }"  
   "return false;"  
   "}"  
}

GetHashCode can also utilize [EqualityComparer](http://msdn2.microsoft.com/en-us/library/ms132123.aspx).

function script:Gen-GetHashCode  
{  
   param ( [int] $count = $(throw "Need a count") )  
   "public override int GetHashCode() {"  
   "int code = 0;"  
   0..($count-1) | %{ "code +=
EqualityComparer<T{0}>.Default.GetHashCode(m_{1});" -f
$upperList[$_],$lowerList[$_] }  
   "return code;"  
   "}"  
}

Both of the operators are likewise straight forward. As before mentioned [EqualityComparer](http://msdn2.microsoft.com/en-us/library/ms132123.aspx) will properly check for null and then perform an Equals call so it can be used as the standard operator code.

function script:Gen-OpEquals  
{  
   param ( [int] $count = $(throw "Need a count") )  
   $OFS = ','  
   $gen = "<" \+ [string](0..($count-1) | %{ "T"+$upperList[$_] }) + ">"  
   "public static bool operator==(Tuple$gen left, Tuple$gen right) {"  
   "return EqualityComparer<Tuple$gen>.Default.Equals(left,right); }"  
   "public static bool operator!=(Tuple$gen left, Tuple$gen right) {"  
   "return !EqualityComparer<Tuple$gen>.Default.Equals(left,right); }"  
}

In addition to the methods, the Tuple class generation must be changed to implement IEquatable<Tuple<».

Some will notice that the implementation forces the equality comparison to be against a Tuple vs an ITuple. There are a couple of reasons for this.

I have come up against specific scenarios where I wanted to compare Tuple but not ITuple. This is not saying they don't exist (they do). But I prefer to leave an implementation until I find a justification for implementing it.
By constraining to IEquatable<Tuple> we are always comparing apples to apples. If you try and perform an Equals against ITuple you're leaving yourself open to comparing apples and oranges. Since ITuple<TA,TB> implements ITuple it is a valid target for the overload. This type of equality seems scenario dependent and as such I left it out for the time. Note with our current implementation it would be very easy to come back and add this later.
To make #2 even stranger, once MutableTuples are implemented an implemantation of IEquatable<ITuple> might actually be comparing Tuple<TA,TB,TC> to MutableTuple.

A Smarter Select Stringrecurse

2008-01-21T00:00:00+00:00

Previously I blogged about a recursive select-string function. Recently I’ve extended it a bit. I found the function to be very useful but when I encountered problems searching large directories that contained binary files. Namely searching them usually returned a result of sorts and printing out the contents of a binary file caused my console to beep in a rather annoying fashion. To fix this I added a new parameter that will perform a slightly smarter search by filtering out binary files.

function Select-StringRecurse()  
{  
    param ( [string]$text = $(throw "Need text to search for"),  
            [string[]]$include = "*",  
            [switch]$smart = $false) 

    $smartRegex = "^\\.(lib|exe|obj|bin|tlb|pdb)$"  
    gci -re -in $include |   
        ? { -not $_.PSIsContainer } |   
        ? { (-not ($smart)) -or (-not ($_.Extension -match $smartRegex)) } |  
        % { write-debug "Considering: $($_.FullName)"; ss $text $_.FullName }  
}  

Clr Memory Model

2008-01-17T00:00:00+00:00

Internally and externally I see a lot of questions about the .Net Memory Model. I think a lot of the confusion comes from the specs. Mainly that there are really two of them.

The first is the ECMA CLI Memory Model (Partition 1, Section 12). This standard introduces a relaxed memory model which, IMHO, makes multi-threading program a bit difficult. For instance it allows for write reordering which can be quite confusing to programmers (and result in very hard to track bugs).

The CLR 2.0 Memory Model is a stricter version of the EMCA model. There are two excellent sources of information on the more strict version.

http://msdn.microsoft.com/msdnmag/issues/05/10/MemoryModels/ - Vance Morrison’s detailed article on multi-threaded apps and locking techniques. He goes into a bit of detail on how the ECMA and CLR 2.0 models differ and the justification for making them do so.
http://www.bluebytesoftware.com/blog/2007/11/10/CLR20MemoryModel.aspx - Joe Duffy sums up Vance’s article and defines a set of 6 simple rules to the memory model.

For anyone doing multi-threaded programming in .Net, both of these articles are a must read.

Mixing Seh And C Exceptions

2008-01-11T00:00:00+00:00

Recently I had a half day adventure trying to catch a SafeIntException in code I was writing. The particular function involved a bit of math with user controlled values. Writing a bunch of IfFailGo’s with several TryAdd style API’s was getting tiresome so I decided to just use SafeInt, catch an overflow and return a failure code.

After recompiling the code I got an error because I hadn’t enabled C++ exception handling in the project. Fix to support synchronous exceptions (/Ehs), recompile. Now I get a different error because a portion of the code base is using SEH exception handling. No problem, changed the project to support asynchronous exception handling as well (/Eha). And yet again another error, the compiler now said that I could not use variables with a destructor in the same method that had an SEH try block.

This put me in a bit of a corner. The code base is littered with SEH exception handling and much of it is perfectly valid. Luckily the number of places using both SEH and destructors was fairly limited. What I needed was a way to unify the exception handling to allow me to use destructors in the same place as SEH exceptions.

With /Eha there is a built-in way. You can catch all SEH exceptions using the C++ syntax “catch (…)”. However this method is limiting because you cannot access the SEH exception code and you get notified after the stack unwind occurs. In SEH an exception filter is run before a stack unwind occurs which makes it infinitely easier to track down your bugs as you can just look down the stack trace and see the line of code that faulted.

After a bit of research I found a solution. I’m chose to blog about it because many of the questions I needed answers for were either 1) undocumented or 2) documented to vaguely. In the end I setup some experiment projects to confirm the behaviors I expected.

The magic function I was looking for was _set_se_translator. It takes a function pointer as an argument and returns the previous value. The documentation states that when a C/SEH exception is raised this function pointer is called once per function that has a C++ try block. It can be used to throw a C++ exception in place of a SEH exception. So you can now translate a SEH exception into a C++ one.

What the documentation didn’t explain (and I was very curious about) is how this interweaves with existing SEH __try/__except/__finally blocks. After some experimentation I discovered that it works extremely well. SEH exceptions occur in two phases

Process the exception filters until either EXCEPTION_EXECUTE_HANDLER or EXCEPTION_CONTINUE_EXECUTION is returned
If EXECEPTION_EXECUTE_HANDLER is returned the stack is unwound to the point of the exception handler block and it is then executed.

The value passed to _set_se_translator participates in phase 1 of the process. As the CRT is looking down the stack for a SEH exception filter, if it finds a C++ try block it will also process the last value passed to _set_se_translator and allow it to throw a C++ exception. If an exception is thrown the stack is unwound to that point and then the exception is thrown. This means that if you call into other code which uses traditional SEH handlers they will operate just as they did before you started using _set_se_translator. Most importantly __finally blocks will run and __except blocks above you in the stack still process and can handle SEH exceptions.

The next step was to write an easy to use wrapper for this functionality. I used two classes to implement this approach.

The first class is designed to properly install a translator at a given point in the stack and ensure that it is reset to the previous value when that stack frame is popped off.

extern void SehTranslatorFunction(unsigned int, struct _EXCEPTION_POINTERS*);

class SehGuard
{
public:
    SehGuard()
    {
        m_prev = _set_se_translator(SehTranslatorFunction);
    }

    ~SehGuard()
    {
        _set_se_translator(m_prev);
    }

private:
    _se_translator_function m_prev;
};

The second part is to actually throw an exception inside of SehTranslatorFunction. Also to add an assert so that when an SEH exception is produced I can break at the point of failure (as opposed to in the catch block where the stack will be unwound.

class SehException
{
public:
    SehException(int code) : m_code(code) { }
private:
    unsigned int m_code;
};
void SehTranslatorFunction(unsigned int code, struct _EXCEPTION_POINTERS*)
{
    MyAssertFunction(false,"Caught an SEH exception");
    throw SehException(code);
}

Now whenever I hit a point where I want to guard against SEH exceptions, I just put and instance of SehGuard on the stack and catch SehException instances. No traditional SEH needed.

Tuples Part 4 Interface

2008-01-10T00:00:00+00:00

Now we have a decent tuple generation script which produces a very usable set of tuple classes. After awhile I ended up getting stuck because the tuples are not flexible enough. It’s not possible to use a 2 pair tuple where a 1 pair is expected even though it meets the requirements.

Process(Tuple.Create("foo"));
Process(Tuple.Create("foo", 42));

public static void Process<TA>(ITuple<TA> tuple)
{


}

I considered two approaches to this problem; inheritance and interface. I debated the inheritance one for awhile. I couldn’t convince myself one way or another if a Tuple<int,int> was a Tuple or merely behaved like one. Also once we introduce a MutableTuple class inheritance won't fix the problem (unless you introduce nasty shadowing variables). Instead I opted for an interface based approach.

In addition to defining the basic interface I added two methods to the base most interface. These methods allow methods to operate on tuples in generic ways regardless of the pair count.

int Count { get; }
object this[int index] { get; }

Generating the implementation is straight forward at this point considering the past solutions. You’ll also have to alter the class definition to inherit from the appropriate ITuple interface.

Hopefully by now it’s becoming clear why having a script to regenerate the large code base is a good idea. It’s easy to make sweeping changes to your implementation.

    function script:Gen-ITuple  
    {  
        param ( [int] $count = $(throw "Need a count") )   
        $OFS = ','   
        $gen = "<" + [string](0..($count-1) | %{ "T"+$upperList[$_] }) + ">"       
        $base = ""   
        if ( $count -ne 1 )   
        {   
            $baseGen = "<" + [string](0..($count-2) | %{ "T"+$upperList[$_] }) + ">"       
            $base = ": ITuple$baseGen"   
        }   
        else   
        {   
            $base = ": ITuple"   
        }   
        "public interface ITuple$gen $base {"   
        "T{0} {0} {{ get; }}" -f $upperList[$count-1]    
        "}"   
    }

    function script:Gen-TupleAccess  
    {  
        param ( [int] $count = $(throw "Need a count") )   
        "public int Count { get { return $count; } }"   
        "public object this[int index] { get { switch (index){ "   
        0..($count-1) | %{ "case $($_): return m_$($lowerList[$_]);" }   
        "default: throw new InvalidOperationException(""Bad Index"");"   
        "} } }"   
    }

Tuples Part 3 Type Inference Friendly Constructor

2008-01-07T00:00:00+00:00

Last time we were left with a constructor that required us to explicitly specify generic parameters. This is not always easy or possible. We’ll now alter the script to generate a constructor which utilizes type inference to create a Tuple. In addition, all tuples will use the same overloaded method making the creation uniform.

The best way to use type inference to create a generic argument is through static methods. In C# and VB it’s legal to define a non-generic class with the same name as a generic class. I tend to create a non-generic class with a static Create method that takes advantage of type inference. For tuples the method will look like the following.

public partial class Tuple
{
    public static Tuple<TA> Create<TA>(TA a)
    { 
        return new Tuple<TA>(a); 
    }
}

This allows us to write the following code.

var tuple = Tuple.Create("foo");

Partial classes are used because we will be generating one per Tuple class that we create. It’s just easier to script it this way.

The method is very straight forward. We need one new additional string for the arguments to the constructor. It’s created along the same line as the previous strings.

function script:Gen-InferenceConstructor  
{  
    param ( [int] $count = $(throw "Need a count") )   
    $OFS = ','   
    $gen = "<" + [string](0..($count-1) | %{ "T"+$upperList[$_] }) + ">"       
    $list = [string](0..$($count-1) | %{ "T{0} {1}" -f $upperList[$_],$lowerList[$_] })   
    $argList = [string](0..$($count-1) | %{ $lowerList[$_] })   
    "public partial class Tuple {"   
    "public static Tuple$gen Create$gen($list) { return new Tuple$gen($argList); } "   
    "}"   
}

Now just add a call to this function in Get-Tuple and the code is now inference friendly.

Next up is defining an interface for tuples that will allow us to treat a Tuple<2> as a Tuple<1>. Both have an “A” property and should be able to be used in a generic way.

Isynchronizeinvoke Now

2008-01-07T00:00:00+00:00

ISynchronizeInvoke is an interface which allows you to execute a delegate synchronously or asynchronously. The implementer of the interface can control how the delegate is executed. In particular the implementer controls on which thread the delegate is executed. It’s common for thread sensitive objects to implement this interface. It allows consumers to execute long lived actions on a background thread and then use the ISynchronizeInvoke interface to get back onto the original thread. System.Windows.Forms.Control is a great example of this usage pattern.

Several API’s that I own are very thread aware and will often defer to a background thread for the completion of their operation. It makes it easier to build UI on top of it. They take an ISynchronizedInvoke as a parameter in order to properly signal the caller than an operation is completed.

The difficulty can come in testing it. Many implementers of ISynchronizeInvoke use message pumping to implement the behavior. As a result it’s not always easy to use for testing (unit testing in particular). To work around this I designed an implementation of ISynchronizeInvoke that does not rely on the message pumping but provides a completely compliant ISynchronizeInvoke implementation.

The idea is to just do it … now. I call the class ImmediateInvoke. The basic methods are straight forward.

object ISynchronizeInvoke.Invoke(Delegate method, object[] args)
{
    return method.DynamicInvoke(args);
}

bool ISynchronizeInvoke.InvokeRequired
{
    get { return false; }
}

The other two methods are a little more tricky. The require an implementation of IAsyncResult. The basic usage pattern is the consumer will call BeginInvoke, peform some operations and finally call EndInvoke when it wants to join with the asynchronous operation. I will use the thread pool to perform this operation and define a private nested class for the IAsnycResult implementation called AsyncResult

The implementation needs a few variables to implement the contract.

m_handle - An implementation of ManualResetEvent to satisfy the AsyncWaitHandle property
m_completed - A simple int to capture whether or not we have completed
m_return - Return of the delegate
m_exception - Exception thrown by calling the delegate.

Most of the properties are straight forward.

public object AsyncState
{
    get { return this; }
}

public System.Threading.WaitHandle AsyncWaitHandle
{
    get { return m_handle; }
}

public bool CompletedSynchronously
{
    get { return false; }
}

public bool IsCompleted
{
    get { return m_completed == 1; }
}

Now we need to define a method to run the delegate passed to BeginInvoke in the thread pool and update the state as we go along. I call this method directly from the constructor.

private void RunDelegateAsync(Delegate method, object[] args)
{
    WaitCallback del = delegate(object unused)
    {
        try
        {
            object temp = method.DynamicInvoke(args);
            Interlocked.Exchange(ref m_return, temp);
        }
        catch (Exception ex)
        {
            Interlocked.Exchange(ref m_exception, ex);
        }

        Interlocked.Exchange(ref m_completed, 1);
        m_handle.Set();
    };

    ThreadPool.QueueUserWorkItem(del);
}

Notice I’ve avoided using a lock in this implementation. This is safe for this case. All of the members are set atomically. Only m_completed can be accessed before the operation is completed and it is simply checked for the value 1. Since the value is set atomically this is safe. In the implementation of EndInvoke I will not access any of the other variables until the WaitHandle is signaled and then I will not make any decision based on the contents of their values (rather the abscence or presence).

Also notice that I did not explicitly dispose of the WaitHandle. This is a quirk of the ISynchronizeInvoke interface. It specifies that callers of BeginInvoke must call EndInvoke and that the IAsyncResult must be valid until EndInvoke is called. As such you can’t really free a resource inside of the IAsyncResult implementation. In fact if you implement IDisposable, who will see it (generally not possible since C# and VB don’t support covariant return types). Instead you should free it as part of your EndInvoke implementation.

Now BeginInvoke and EndInvoke.

IAsyncResult ISynchronizeInvoke.BeginInvoke(Delegate method, object[] args)
{
    return new AsyncResult(method, args);
}

object ISynchronizeInvoke.EndInvoke(IAsyncResult result)
{
    var r = (AsyncResult)result;
    try
    {
        r.AsyncWaitHandle.WaitOne();
    }
    finally
    {
        r.AsyncWaitHandle.Close();
    }

    if (r.m_exception != null)
    {
        throw new Exception("Error during BeginInvoke", r.m_exception);
    }
    return r.m_return;
}

Unfortunately EndInvoke has to take care of two cases. The first is the delegate completed successfully and produced a value which can now be returned as a part of the interface contract. The other case is when the delegate throws and exception and it’s a bit more tricky. The exception cannot be simply re-thrown because you will loose all of the original call stack and generally speaking most of the data which would help track down the problem. The better option is to throw a new exception and make this exception the inner exception.

This sample could be improved in a few ways (delay create the WaitHandle, rethrow with something other than System.Exception). But it’s a compliant version that gets the job done.

Tuples Part 2 Basic Structure

2008-01-04T00:00:00+00:00

Part 1 of the series outlined the basic structure of the tuple. This entry will produce a PowerShell script that will generate N tuple classes containing 1-N name value pairs.

The first step is to get a few script variables defined. All of the names used in the tuples will be lower and upper case single characters tied to a specific index. To make the script a bit shorter we will define indexable arrays up front for the letters. We’ll also grad the tuple count.

$script:tupleCount = [int]$args[0] 
$script:lowerList = 0..25 | %{ [char]([int][char]'a'+$_) } 
$script:upperList = 0..25 | %{ [char]([int][char]'A'+$_) }

All of the functions in this script will output an array of strings into the PowerShell pipeline. One of the neat/confusing features of PowerShell is that values that are not directly used in a function are passed onto the pipeline. This will conveniently allow us to type in literal code and hopefully increase readability.

Now for the function that generates a property. The tuples will be immutable by default as such we will generate private read only fields and simple getters.

It would also be just as plausible to skip the property here and instead produce a read only public field. Later on we will be altering the tuples to be used in a generic fashion through interfaces. Interfaces cannot define fields and instead we will need properties.

 
function script:Gen-Property 
{ 
    param ( [int] $index  = $(throw "Need an index") ) 
@" 
    private readonly T{0} m_{1}; 
    public T{0} {0} {{ get {{ return m_{1}; }} }}

"@ -f $upperList[$index],$lowerList[$index] 
}

Next up we need to define a constructor. All of the fields in the tuple are read only so we must define a constructor for the consumer (otherwise the tuples would be useless).

Generating the parameter list string would be tedious in most languages but PowerShell makes it a snap. When converting an array of strings into a single string the individual strings will be combined with the value of the *** $OFS variable (default is space). We can switch this to a comma and provide a quick pipeline for the parameter list.

function script:Gen-Constructor 
{ 
    param ( [int] $count = $(throw "Need a count") ) 
    $OFS = ',' 
    $list = [string](0..$($count-1) | %{ "T{0} {1}" -f $upperList[$_],$lowerList[$_] }) 
    "public Tuple($list) {" 
    0..($count-1) | %{ "m_{0} = {0};" -f $lowerList[$_] } 
    "}" 
}

Now that we have the basics we can generate the class. We’ll use the same $OFS trick to generate the generic argument list here.

function script:Get-Tuple 
{ 
    param ( [int] $count ) 
    $OFS = ',' 
    $gen = "<" + [string](0..($count-1) | %{ "T"+$upperList[$_] }) + ">"    

    "public sealed class Tuple$gen {" 
    0..($count-1) | %{ Gen-Property $_ } 
    Gen-Constructor $count 
    "}" 
}

Now all that’s left is processing the arguments

[string](0..($tupleCount-1) | %{ Get-Tuple ($_+1) })

Next step is to generate code that is more type inference friendly.

Tuples Part 1

2008-01-03T00:00:00+00:00

A tuple in computer science can be described as a set of name/value pairs. In some cases it can be described as simply a set of values that are accessible via an index ¹. Previously I discussed how to create a Tuple inside of PowerShell. This series will focus on the use of Tuples in DotNet and how to use PowerShell to generate DotNet code.

This series will also distinguish between mutable and immutable tuples. As DotNet is shifting it’s focus on parallel programming, immutable types are becoming more important. Therefore this serious will focus on Tuples as immutable types and later examine mutable tuples.

In Visual Studio 2005, both C# and VB acquired tuples as a part of the programming language in the form of Anonymous Types. These fit all of the properties of a tuple. The one difference is in VB, anonymous types are mutable by default. This can be changed though by using the Key keyword.

However anonymous types are lacking one quality which severely lessens their usefulness. Their type cannot be described. This prevents them from being used as parameters, fields, generic parameters etc … Unless you use late binding or terribly awkward casts this is limiting.

To get around this, we will be defining a set of generic tuple class supporting 1-N name value pairs. The great downside is because we will be predefining these types the names in the name value pair will be fixed. We will be using A-N for 1-N pairs.

This is very limiting in itself because it’s reducing the expressiveness of a type. Anonymous types are much more expressive since they have names. Now types will have properties A,B, etc …

For me this still works. In my code, I only end up using tuples when I need to pass data around between tightly coupled classes, or just within the same class. Since the creation and use are so close loosing the full expressiveness of the name is not that limiting.

In addition, our tuple implementation will leverage type inference as much as possible such that the following code can be written.

var tuple = Tuple.Create("foo");
Console.WriteLine(tuple.A);

Why write a script to generate these classes? Wouldn’t it just be easier to just do this by hand’ Yes and no. If you are doing a fixed set of short used classes then yes, do it by hand. These scripts evolved out of my use of tuples. Once I would settle on a structure and I would think of a new feature I needed. Typically I have tuples defined up to 5 fields. Retyping out a new feature got tiresome and error prone. With a scripting solution I could add a new feature and tests in just a few minutes. The series is very representative of the way my solution changed over time. Simple at first but I added features as the situation dictated. Having a scripting solution saved me a lot of time.

Next up, generating the basic structure.

In this case, the index just becomes the name and hence a name/value pair. ↩

Ternary Reducing Code

2007-12-18T00:00:00+00:00

I like my scripts to be readable and terse. They’re scripts after all and I want to get the most done with the least amount of code. There’s a lot to be said for having a readable script but I only value that when I intend to keep the script around for awhile.

PowerShell does not have a ternary operator and that often frustrates me as I end up writing lots of verbose code.

Once again, fix it by introducing a small function into my profile. Not quite a true ternary operator because it evaluates both of the result arguments. But it does the trick for most situations.

function Get-Ternary()  
{  
    param ( [bool]$condition = $(throw "Need a conditional"),  
            $valueTrue = $(throw "Need a value for the true condition"),  
            $valueFalse = $(throw "Need a value for the false condition") )  
    if ( $condition )  
    {  
        return $valueTrue  
    }  
    else  
    {  
        return $valueFalse  
    }  
}

Now I can type

PS> Get-Ternary $cond 1 42

C Lambda Type Inference

2007-12-14T00:00:00+00:00

One of the limitations of C# type inference is that you cannot use it to infer the type of a lambda expression. For example, the following code will not compile

var f = () => 4;

Normally this is not too much of an issue because you can just explicitly declare the type of the lambda expression.

Func<int> f = () => 4;

However, this can be annoying at times. Once you start defining complex lambda expressions the Func/Action declaration can be quite convoluted. Even worse, if your lambda returns an anonymous type, there is no way to declare a Func<> with the anonymous type parameter because you cannot describe it’s shape.

Func<'> f = () => new { Name = "foo" };

This is fixable though by using type inference. The method is very similar to other anonymous type type tricks. While lambda type inference is not supported for variable declaration it is supported for parameters. C# supports return type inference so that can be used to type the variable.

static Func<T> Lambda<T>(Func<T> del)
{
    return del;
}

static void Main(string[] args)
{
    var f = Lambda(() => new { Name = "foo" });
}

Filtering Get Psdrive To All Local Drives

2007-12-06T00:00:00+00:00

Recently I needed to filter the return of get-psdrive to return all of my local hard drives. I didn’t want to accidentally start operating on floppies, CDROM’s and more importantly, network drives. There are a couple of ways to do this but I found the most straight forward is to combine the WMI data with get-psdrive.

E:\temp> get-wmiobject win32_volume | ? { $_.DriveType -eq 3 } | % { get-
psdrive  
$_.DriveLetter[0] }

Name       Provider      Root
CurrentLocation  
\----       \--------      \----
\---------------  
C          FileSystem    C:\
...nfig\PowerShell  
E          FileSystem    E:\
temp

DriveType is a property of the Win32_Volume structure which enumerates the type of drive. The value 3 stands for Local Disk. Below is the full list of values.

- Unknown  
- No Root Directory  
- Removable Disk  
- Local Disk  
- Network Drive  
- Compact Disk  
- RAM Disk

Types Of Immutability

2007-12-04T00:00:00+00:00

By definition, an immutable object in computer science is one that is not able to change. Parallel coding is becoming more necessary as the number of cores in a processor are increasing but not the overall speed. As such immutability is will become more important because it is an important asset of multithreaded programming. Immutable objects are inherently thread safe.

Like other categorizations in computer science, there are degrees of being unable to change. This post is an attempt to outline and categorize several of the variations.

Note: these are not standard categorizations. They are merely my attempt to name several of the scenarios you can encounter with immutable objects.

Immutable

Object itself has no fields that can change. In addition all of it’s fields are also Immutable and thus cannot be changed. This object is carved in stone and short of process corruption, it will be the same in every way.

For primitives excluding string this can be ensured by making them read only (.initonly in IL). For reference types the field must be read only and the type must also meet the Immutable guidelines.

Types with meet the Immutable guarantee are inherently thread safe. They are not subject to race conditions because they cannot be changed and thus viewed differently between threads.

Shallow Immutable

The direct contents of this object must be immutable in the sense that they cannot be changed to point to another object. However there is no guarantee that all of the fields are themselves immutable.

All of the fields in the object must be read only. For primitive values this is enough to guarantee them meeting the Immutable guidelines and hence Shallow Immutable. For reference types this will ensure they will point to the same object thus doing all that is needed to meet the Shallow Immutable Guarantee.

Types with this guarantee are thread safe to a degree. They are thread safe as long as you are accessing the fields with meet the Immutable guarantee. You can also access the references which are read only as long as you do not call any instance methods. For instance a Null check is thread safe.

Shallow Safe Immutable

Slightly stronger guarantee than Shallow Immutable. For all fields that are read only but not immutable, the type must be thread safe.

Types which meet this guarantee are thread safe also to a degree. They are as thread safe as the fields which are guaranteed to be thread safe.

Getting My Fortune

2007-12-03T00:00:00+00:00

Fortune is a Unix command that gets a random message from a set of databases and displays it on the screen. These messages have a wide variety but tend to be funny, quirky or famous quotes (most are indeed geeky).

Nearly all unix systems have a version of Fortune. Windows doesn’t have any version by default. It provides no real functionality other than humor and amusement but it’s something I miss. Most users add it to their .profile script so they get a new fortune every time they log in.

There are a couple of heavy weight options to get fortune on my machine?? but I prefer something a bit lighter. I did a quick search and discovered that Doug Hughes implemented a fortune web service. This is just above as light weight as it gets so I implemented a simple PSCmdlet, get-fortune, to do the work (code below).

Now I can just add a quick get-fortune to my profile script.

C:\Users\jaredp> get-fortune pets

Does the name Pavlov ring a bell?

<Cmdlet(VerbsCommon.Get, "Fortune")> _
Public Class GetFortune
    Inherits PSCmdlet

    Public Const TopicHelpMessage = "Restricts fortune output to the specified topic"

    Private m_topic As String = String.Empty
    Private m_minimumLength As Integer
    Private m_maximumLength As Integer
    Private m_timeout As Integer = 100000

    <Parameter( _
               Position:=0, _
               HelpMessage:=TopicHelpMessage, _
               ValueFromPipeline:=True, _
               ValueFromPipelineByPropertyName:=True)> _
    Public Property Topic() As String
        Get
            Return m_topic
        End Get
        Set(ByVal value As String)
            m_topic = value
        End Set
    End Property

    <Parameter(Position:=1)> _
    Public Property MinimumLength() As Integer
        Get
            Return m_minimumLength
        End Get
        Set(ByVal value As Integer)
            m_minimumLength = value
        End Set
    End Property

    <Parameter(Position:=2)> _
    Public Property MaximumLength() As Integer
        Get
            Return m_maximumLength
        End Get
        Set(ByVal value As Integer)
            m_maximumLength = value
        End Set
    End Property

    <Parameter(Position:=3)> _
    Public Property Timeout() As Integer
        Get
            Return m_timeout
        End Get
        Set(ByVal value As Integer)
            m_timeout = value
        End Set
    End Property

    Protected Overrides Sub ProcessRecord()
        Dim proxy As New DougHughes.fortune
        Dim topic = m_topic
        If topic Is Nothing Then
            topic = String.Empty
        End If

        proxy.Timeout = m_timeout
        WriteObject(proxy.getFortune(topic, m_minimumLength, m_maximumLength))
    End Sub

End Class

Viemu And Suites

2007-12-02T00:00:00+00:00

ViEmu is a Visual Studio Package which adds Vim keybinding support into Visual Studio. For former VI users this is huge benefit as I can use all of my cryptic key combinations inside of Visual Studio.

For those unfamiliar, Vi is one of the original editors for the Unix operating systems. Vim (VI iMproved) is a set of improvements on the original Vi editor. At it’s core Vi/Vim is a modal editor which means that it has multiple modes of input. Mainly there are

Command - allows you to enter commands to the editor
Edit -modifying the text

Getting started with Vi/Vim is quite a challenge because the learning curve is incredibly steep. Once you get it down though you can accomplish a whole lot more with just a few key strokes. ViEmu brings this power into Visual Studio.

The only downside is that ViEmu dramatically alters the way keystrokes affect the environment. This is a really interferes when I need to run code suites on my machine as it will cause any that use keystrokes to fail.

Again, PowerShell is the answer. I just disable ViEmu while suites are running on my machine and quickly re-enable them afterwards.

function Enable-ViEmu()  
{  
   sp "hkcu:\software\microsoft\VisualStudio\9.0\ViEmu" "Enable" 1  
   sp "hkcu:\software\microsoft\VisualStudio\9.0\ViEmu" "AllowKbdClashes" 0  
}

function Disable-ViEmu()  
{  
   sp "hkcu:\software\microsoft\VisualStudio\9.0\ViEmu" "Enable" 0  
   sp "hkcu:\software\microsoft\VisualStudio\9.0\ViEmu" "AllowKbdClashes" 1  
}

ViEmu provides this UI but it’s a mode switch. I tend to run my suites from the command line and I find opening the UI and disabling takes too much time and is a distraction. Running a quick disable-viemu takes virtually no time and fits right into the command line.

Calling Extension Methods On Null Objects

2007-11-30T00:00:00+00:00

One of the gotchas for Extension Methods is that it’s legal to call them on Null References. This isn’t really surprising when you think about the feature. Boiled down to a fundamental level, extension methods are just syntactic sugar for calling a static method and automatically passing the first parameter ¹. However it can catch the unwary off guard.

The two items that are a little bit different between calling an Instance vs Extension method on a null reference is

The exception (if thrown) will be at the method site instead of the call site. Not really an issue because you can just jump down the stack frame.
There may not be an exception thrown. As long as you don’t actually use the extension method target, the code will not necessary throw². For instance consider the following code

<Extension()> _
Public Function IsNothing(ByVal o As Object) As Boolean
    Return o Is Nothing
End Function


Sub Test()
    Dim x As String = Nothing
    Dim b = x.IsNothing()   ' b = True
End Sub

This is legal and will not throw. However I don’t recomend that you write it.

Then again you could also claim that instance methods are just syntactic sugar for calling static methods without having to pass this/Me as the first parameter. ↩
In some ways this is similar to C++. C++ doesn’t do NULL reference checking automatically (it waits for you to access data on a NULL reference and then crashes if you’re lucky). If you call a method on a NULL pointer but don’t actually access any member variables, it will likely run fine. ↩

Tuples In Powershell

2007-11-29T00:00:00+00:00

Tuples in computer science are usually light weight record objects with simple name value pairs. In scripting languages it is very handy to create them on the fly. For quite some time I was using associative arrays in PowerShell to do just that.

PS>$a = @{Name="MyName";Value="MyValue"}

It has essentially everything you would need from a tuple inside of a scripting language. The more and more I use PowerShell though I’ve found that this is not always a good idea. It comes back to PowerShell pipelining. Whenever you pass a collection to a pipeline PowerShell will unroll the collection and pass the individual items.

Under the hood, an associative array is a System.Hashtable. As a result it is a collection of name value pairs. Hence when you pass this tuple through a pipeline, it is torn apart and each element of the tuple is passed as a separate object.

PS>$a = @{Name1="Value1";Name2="Value2"}
PS>$a

Name                           Value
----                           -----
Name2                          Value2
Name1                          Value1

Hence I’ve now taken a new route. Create an actual tuple :)

function New-Tuple()  
{  
    param ( [object[]]$list= $(throw "Please specify the list of names and values") ) 

    $tuple = new-object psobject  
    for ( $i= 0 ; $i -lt $list.Length; $i = $i+2)  
    {  
        $name = [string]($list[$i])  
        $value = $list[$i+1]  
        $tuple | add-member NoteProperty $name $value  
    } 

    return $tuple  
}

PS>$a = New Tuple Name,1,Value,2

The result can now be passed around pipelines as a single entity.

Piping Elements Of A String

2007-11-28T00:00:00+00:00

Quick script that will allow you to pipe each char in a String into the PowerShell pipeline.

function PipeStringChar()
{
    param ( [string]$toPipe )
    for ( $i = 0; $i -lt $toPipe.Length; $i++ )
    {
        write-output $toPipe[$i]
    }
}

Alternatively you can do this by using the ToCharArray method. However this will create a new array in memory and if you have a large string that will be fairly expensive. The above method will do it the PowerShell way.

Type Inference And Ienumerable

2007-11-26T00:00:00+00:00

This is somewhat of a follow up on a previous post I did on the difference between IEnumerable(Of T) and the IEnumerable interfaces.

I’ve seen several people type in the following code and wonder if there was a fundamental bug in the type inference code.

Private Sub Form1_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load
    For Each cur In Controls
        cur.Text = "A Value"
    Next
End Sub

This code will produce an error stating that “Text” is not a member of object. Users expected type inference to type the variable “cur” as Control. Unfortunately this is “By Design”.

Much of the original .Net Framework was written before the CLR implemented support for generics. As a result all of the collection classes were loosely typed to Object by implementing IEnumerable. So in this case type inference will correctly type this as Object.

There are 2 ways to fix this problem.

Explicitly type the For Each variable to be the actual type of objects in the collection
Use a Shim to change the type of the collection

Autosize And Dockstyle Fill Don T Mix

2007-11-21T00:00:00+00:00

The title of this post essentially says it all. AutoSize and DockStyle.Fill don’t mix well together. Both properties exist to describe the size relationship relative to the rest of the control but they do so in conflicting ways.

AutoSize is a property describing the size of a control relative to it’s contents. Setting this to True will generally cause a control to resize itself so that it takes up only enough room to display it’s contents.

DockStyle.Fill is a property describing the size of a control relative to the size of it’s container. A control will resize to fit all of the empty space in it’s parent container with this property set.

In effect these properties represent opposite ways of describing size. Most controls will prevent you from setting conflicting settings in the designer. However it’s difficult to prevent you from setting conflicting settings between a container and it’s contents.

For instance, one combination is the following. This may seem a little odd at first. It commonly happens when you are trying to embed a non-autosizing control inside a Container that supports AutoSize. For instance if you are trying to place a TextBox inside of a FlowLayoutPanel

Container
- AutoSize = true
- AutoSizeMode = GrowAndShrink
Containee
- Dock = DockStyle.Fill

If you try this code it will often cause the Container Control to shrink to nothing. Why’ In affect both controls are asking each other how big they should be and with no one having the deciding factor they agree on … nothing.

Instead for this scenario you should leave the Containee to DockStyle.None and manually set the size.

Vs2008 Ships

2007-11-19T00:00:00+00:00

Orcas has shipped :)

http://msdn2.microsoft.com/en-us/vstudio/default.aspx

Lambda Unexpected Behavior

2007-11-19T00:00:00+00:00

One item you strive to avoid when you design and implement a feature is unexpected behavior. Unfortunately there is one case we couldn’t avoid with Lambda’s in VB9. I just ran into the this problem when coding up a handler. I wanted to disable a button when the text of particular TextBox was empty. I wrote the following code to handle the situation.

AddHandler c.TextChanged, Function() okButton.Enabled = (0 <> c.Text.Length)

This doesn’t quite do what I intended. This instead will simply compare the two values.

In VB9 Lambda Expressions are always an expression. In version 9 of VB, there is no concept of an assignment as an expression. There is only a statement version. As a result this doesn’t do anything useful.

This has tripped up a few people along the way. It’s an unfortunate side effect of only supporting expression lambdas.

I was able to work around this by defining a function which did what I intended. I called this function in the lambda expression and the problem was solved.

Extension Methods Without 3 5 Framework

2007-11-16T00:00:00+00:00

For a time I’ve been avoiding extension methods. Not because I’m opposed to using them but because of the 3.5 Framework.

A lot of the tools I own are designed to be very light weight tools that only require the user to have 2.0 installed on their machine. I find that the easier that tools are to install, the more likely people are to use them.

Extension methods require the ExtensionAttribute be available. Since the attribute is declared in a 3.5 Framework assembly it’s not possible to use extension methods without the 3.5 framework. At least, that’s what I thought up until I read an recent MSDN article.

You can simply define the ExtensionAttribute in your assembly and extension methods will start working. No references to the 3.5 framework required. It’s a lightweight solution that adds the full power of extension methods to your program.

Namespace System.Runtime.CompilerServices
    Class ExtensionAttribute
        Inherits Attribute
    End Class
End Namespace

Names Of Anonymous Type Members

2007-11-09T00:00:00+00:00

Recently I was asked how can you get a list of anonymous type member names given a query of anonymous types. The quick answer is that you can use a quick bit of reflection to get back the names.

Public Function GetAnonymousTypeMemberNames(Of T)(ByVal anonymousType As T) As List(Of String)
    Dim type = GetType(T)
    Dim list = New List(Of String)
    For Each cur In type.GetProperties(Reflection.BindingFlags.Public Or Reflection.BindingFlags.Instance)
        list.Add(cur.Name)
    Next
    Return list
End Function

The longer question is how can you get back a list of anonymous type member names given a query result? As long as you know the query will be populated you can just use the first member to get your result.

Dim q = From it In "astring" Select a = it, b = it & "b"
GetAnonymousTypeMemberNames(q.First())

However you can’t guarantee that a query will always have data in it. Another route is to consider the contract of the From … Select statement. This will produce a query which implements IEnumerable(Of T). For a query T will be the anonymous type that is generated¹. We can query the metadata of the returned type to get the System.Type for the anonymous type regardless of the type actually implementing IEnumerable(Of T).

Dim q = From it In "astring" Select a = it, b = it & "b"
Dim enumerableInterface = GetType(IEnumerable(Of ))
Dim enumerableType = q.GetType().GetInterface(enumerableInterface.FullName)
Dim anonymousTypeType = enumerableType.GetGenericArguments(0)

This is assuming that you didn’t write a query which returns a field directly and avoids creating the anonymous type. ↩

C Refactoring Default Booleans And Pointers

2007-10-29T00:00:00+00:00

This is another recount of an experience I had refactoring some C++ code recently. In some ways this is also a follow up of my previous post about why you shouldn’t use boolean parameters (especially default value ones).

I recently refactored a large set of API’s in our code base to remove a common parameter. It was a member variable that was constantly being passed as an argument. For many reasons we decide that this was unacceptable and it was better to use the member variable for instance methods and if needed define static methods which took the parameter as a value.

After making the switch I recompiled and starting running regression suites on the code and found that a whole set were failing. It turns out that many of our methods had the following pattern.

class Student;

int CalculateValue(Student *pStudent, Manager *pManager, bool cache=false);

The parameter I was removing was “pManager”. So now we have a function that used to take 2 or 3 arguments and now takes 1 or 2 arguments. Unfortunately, now all of the places that used to call this function with 2 arguments will now recompile without warning since pointers are implicitly convertible to booleans.

No matter. Redefined the method with the same arguments and did not specify an implementation. This caused compiler errors where necessary and allowed me to fixup the calling code.

Boolean parameters are troublesome. Defaulted value boolean parameters are much more so.

Yes you could attribute this to the fact that I didn’t fully think through a subset of my changes and that’s certainly fair. However if we practiced good coding standards up front, this could have been avoided. If we had originally done any of the following the code would be more resilient to changes.

Chosen an enum instead of a boolean parameter
Forced the caller to specify the boolean parameter by not providing a default

C Placement New Tracking Down A Crash

2007-10-18T00:00:00+00:00

See my previous two posts on an introduction to placement new if you are unfamiliar with the subject.

/2007/10/16/c-new-operator-and-placement-new.html
/2007/10/17/c-placement-new-and-allocators.html

Recently I did a bit of work on the heap management story for our code base. Mainly it was a change to unify the different ways we accessed our internal heap. In the process I unified our operator new story. Part if this involved unifying our placement new allocation overloads. We have several allocators in our code base and I noticed we had several overloads which essentially did the same operation

static inline
void * _cdecl operator new(size_t cbSize, MyCustomAllocator &allocator)
{
    return allocator.Alloc(cbSize);
}

static inline
void * _cdecl operator new(size_t cbSize, MyCustomAllocator *allocator)
{
    return allocator->Alloc(cbSize);
}

This seemed a bit redundant to me so I deleted the one which contained a pointer overload. A while later I tested my changes and our application almost immediately crashed. It only took a few minutes to discover the problem. We had a lot of operations that followed this pattern.

Student *p = new (pAllocator) Student();  // pAllocator typed to MyCustomAllocator*

As I said in my previous posts, new is just another C++ function. As such it participates in overload resolution. This code now binds to the placement new operator and not the placement allocation overload which takes a reference. At first I thought about using a regex search and replace to solve the issue. However I decided this wasn’t a complete solution. There is no guarantee that my regex will catch every case and any case I miss won’t crash until we actually hit the line of code.

The best case scenario here is to turn that line into a compile error. That would guarantee that I fix every location that is a problem. Since new participates in overload resolution you can solve this with a template.

template <typename T>
static inline
void* _cdecl operator new(size_t cbSize, T* allocator)
{
    allocator->ThisWillForceACompileError();
}        

This caused all places where a pointer type was passed to new which wasn’t explicitly “void*” to turn into a compiler error. This quickly outlined all of the places I needed to fix up and guaranteed that my fix didn’t allow developers who were accustomed to using a pointer to the allocator to accidentally introduce a bug into the code base (myself included :)).

C Placement New And Allocators

2007-10-17T00:00:00+00:00

This is a follow up for my previous post about operator new and placement new. This post will discuss the role of adding a custom allocator and using it with new.

It’s handy to use custom allocators in C++. Certain operations can be done more efficiently on a different type of allocator than the rest of your program. It also allows you to add custom heap tracking, debugging, etc … C++ also makes it very easy to use.

The placement new operator already makes it fairly easy to use a custom allocator with your normal C++ syntax. For example, the bellow code snippet displays how to use a custom allocator with the placement new operator. You may or may not have to define the overloaded “new” depending on your code base.

    static inline
    void * _cdecl operator new(size_t cbSize, void* pv)
    {
        return pv;
    }
    
    void SomeProcedure()
    {
        MyCustomAllocator allocator;
        void *memory = allocator.Alloc(sizeof(Student));
        Student *p = new (memory) Student();
    }

However this code is a bit clunky. It requires an extra local variable and an extra sizeof every time you want to use this pattern. You could eliminate the local by placing the allocation call directly within the new () arguments but for longer type names this would make your lines very long.

A more elegant solution is to further overload operator new. Don’t forget it’s just a normal function (with a few restrictions). That being said you can add an overload that takes in a reference to your allocation engine and perform the allocation inline.

static inline
void * _cdecl operator new(size_t cbSize, MyCustomAllocator &allocator)
{
    return allocator.Alloc(cbSize);
}

void SomeOtherProcedure()
{
    MyCustomAllocator allocator;
    // ...
    Student *p = new (allocator) Student();
}

This is much more elegant, removes the need for a local variable and only adds the length of the allocator name to your normal line length.

C New Operator And Placement New

2007-10-16T00:00:00+00:00

Originally I was going to write this article to detail a particular problem I had recently with placement new in C++. A page or two of writing later I decided it would be best to start with an introduction to the “new” operator itself and the in/outs of overloading/replacing it. The C++ new operator really functions as a 2 step process.

Allocate memory - This is accomplished by calling the in scope “new” operator
Construct object in the allocated memory - Essentially the “this” pointer is pointed to the front of the memory and the constructor is called.

You can customize the new operator in your code by defined operator new at a particular scope.

static inline
void * _cdecl operator new(size_t cbSize)
{
 ... // Return cbSize bytes
}

One item a lot of people don’t realize is that “new” is for most purposes a normal C++ function. Like a normal C++ function you can add parameters. Below is a common overload you will see in code. It is typically referred to as placement new.

// Placement new
static inline
void * _cdecl operator new(
    size_t cbSize,
    void * pv)
{
    return pv;
}

The next question is how to pass the “pv” parameter? When you want to pass additional parameters to “new” you can do so by opening up an argument paren block immediately after the new. When attempting to bind the operator “new” in scope, the C++ compiler will match the first passed argument to the second argument of the defined new, second to third and so on.

Student *p = new (NULL) Student();

This will match the second new operator I defined as NULL is compatible with void*. This particular version of new, placement new, is valuable because it lets you customize how memory is allocated for a particular operation but still us standard C++ constructor syntax. When you redefine operator new you are only taking care of the first step in the C++ new process; memory allocation. The compiler will still perform step #2 on the memory returned from new. This allows the following pattern.

void *p = myAllocator.Allocate(sizeof(Student));
Student *p = new (p) Student();

The second line will bind to the placement new operator I defined above and will simply return the value of “p”. Student will then be constructed in this memory.

Next time I’ll dive into how to make the placement new operator make a custom allocation pattern easier.

Debugging Powershell

2007-10-15T00:00:00+00:00

Debugging PowerShell can be extremely frustrating because it often turns into a session of debugging your own thought process. Often when I hit a PowerShell script issue I find myself feeling like everything is right and I’m just missing something basic. IMHO, this is because I spend the majority of my day programming in compiled languages and don’t completely come out of that box when I’m programming in PowerShell.

For instance the other day I added one of Lee Holme’s scripts to my default profile. The script is used to create a Generic object inside of PowerShell.

http://www.leeholmes.com/blog/CreatingGenericTypesInPowerShell.aspx

The syntax is very clean. It is the same as new-object but requires the additional type parameters.

$PS>new-genericobject Collections.Generic.List int
$PS>

The only problem was I couldn’t get the script to return an object. I tried a couple of operations like passing the output to get-member and seeing what exactly I was creating. I kept getting errors like “No object has been specified …”. My mind kept saying “It’s there, it’s non-null, why doesn’t it have any members!!!”.

Frustrated I turned to some internal aliases and eventually Lee helped me out. Nothing above is in error. The problem is PowerShell will unroll collection classes. In this case I’ve created an empty collection so there is nothing to output or pass to get-member. My issues is that I kept thinking it terms of compiled languages instead of PowerShell’s collection unrolling semantics (which I will add, I quite like).

Sadly any other object creating would have quickly lead me to this solution

C:\Users\jaredpar\winconfig\PowerShell> new-genericobject collections.generic.Ke
yValuePair int,int

                                    Key                                   Value
                                    ---                                   -----
                                      0                                       0

Another way to figure out this problem would be to bypass the collection unrolling in pipelines and directly specify the object to get-member.

C:\Users\jaredpar\winconfig\PowerShell> gm -inputobject $col
   TypeName: System.Collections.Generic.List`1[[System.Int32, mscorlib, Version
=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]]
...

Select Stringrecurse

2007-10-08T00:00:00+00:00

When you need to search for text in a file, select-string is your best friend. It has most of the functionality of old unix grep. In addition it does full regular expression support.

The only downside is that it will only run on files in the current directory. Unlike get-childitem, it has no recurse parameter. In addition, I’m lazy and I don’t like typing out select-string every time. I have the following definitions in my Profile to make this easier.

function Select-StringRecurse()  
{  
'' param ( $text = $(throw "Need text to search for"),  
'''''''? $filter = "*" )  
'' gci -re -in $filter | ? { -not $_.PSIsContainer } | % { ss $text $_ }  
}

set-alias ss'' select-string  
set-alias ssr'? Select-StringRecurse

Ienumerable And Ienumerable Of T 2

2007-10-05T00:00:00+00:00

Quick follow up to my earlier post. Fixing this issue in C# is even easier because of the existence of iterators.

public static IEnumerable<object> Shim(System.Collections.IEnumerable enumerable)
{
    foreach (var cur in enumerable)
    {
        yield return cur;
    }
}

Ienumerable And Ienumerable Of T

2007-10-04T00:00:00+00:00

IEnumerable(Of T) is a huge step up in the 2.0 framework from the original IEnumerable interface. It provides a typed enumeration which eliminates lots of nasty casts. The best part is that IEnumerable(Of T) is backwards compatible with IEnumerable (it inherits from it).

What’s frustrating is that IEnumerable is not forwards compatible with IEnumerable(Of Object). This prevents you from using IEnumerable anywhere IEnumerable(Of Object) or an inferred IEnumerable is expected even though they have almost the same interface. The framework couldn't take the same approach and have IEnumerable inherit IEnumerable(Of Object) because it would have broken any implementer on upgrade to the 2.0 framework.

The good news is this is easy to shim. Since IEnumerable(Of Object) and IEnumerable have virtually the same, it’s easy to create a wrapper class that forwards the calls into the base enumerator.

The basic shim involves two classes

EnumerableShim - Implements IEnumerable(Of Object). This class wraps an IEnumerable object only has two methods.

        Public Function GetEnumerator() As System.Collections.Generic.IEnumerator(Of Object) Implements System.Collections.Generic.IEnumerable(Of Object).GetEnumerator
            Return New EnumeratorShim(m_enumerable.GetEnumerator())
        End Function
    
        Public Function GetEnumerator1() As System.Collections.IEnumerator Implements System.Collections.IEnumerable.GetEnumerator
            Return m_enumerable.GetEnumerator()
        End Function

EnumeratorShim - Implements IEnumerator(Of Object). This class wraps the IEnumerator object created above and implements the standard methods by forwarding all calls to the IEnumerator implementation. For example …

Public ReadOnly Property Current() As Object Implements System.Collections.Generic.IEnumerator(Of Object).Current
    Get
        Return m_impl.Current
    End Get
End Property

Now whenever your stuck with and old framework collection that was not updated for generic support, you can use it in generic situations with a single indirection call.

Dim list As New ArrayList
...
SomeMethod(New EnumerableShim(list))

Net Framework Source Code

2007-10-03T00:00:00+00:00

If you haven’t heard the news yet, you can read the full article on ScottGu’s Blog.

In summary Microsoft has released the source code for the .Net Framework in such a way that you can step into the Framework while debugging. IMHO this is great for users because it allows us to get a lot of insight into parts of the API that are tricky or not well documented.

Take a look at his blog for the full details of hooking this up into the debugger.

Casting To An Anonymous Type

2007-10-01T00:00:00+00:00

This discussion is building upon a previous post on how to acquire an anonymous type … type.

The next question is, how can you cast an arbitrary object into an anonymous type? At a glance this doesn’t seem possible as you cannot directly express the type of an anonymous type in code. For instance the following code is not legal.

Dim o As Object = SomeCall() 
Dim x = Directcast(o, New With {.a = 5})

Even though it’s not possible to directly express the type in code, we can indirectly express it via type inference. Anonymous Types are strongly typed in the compiler and it is possible to infer these types where inference is available.

Public Function CastSpecial(Of T)(ByVal o As Object, ByVal t As T) As T 
    Return DirectCast(o, T)
End Function 

The above function allows us to perform a DirectCast as long as we have an instance of the type we want to cast to. Now we can call the code and passing in the appropriate anonymous type instance.

 Dim o As Object = SomeCall()
 Dim x = CastSpecial(o, New With {.a = 5})

There are a couple of caveats to this approach though.

The actual anonymous type and the one you are casting to must be exactly the same. No polymorphism can be involved.
The anonymous type must be created in the same assembly where you are performing the cast. Anonymous types are specific to the assembly they are created in so performing the cast accross assemblies will cause a InvalidCastException to occur.
It does needlessly create an object to perform the cast.

Customzing Displays In The Debugger For System Types

2007-09-28T00:00:00+00:00

We’ve heard feedback from several customers regarding the way certain types are displayed in the Debugger. Many of the displays exist to maintain the user experience between versions of Visual Studio. We constantly evaluate if this is the correct choice for a given version of the product.

Starting with VS2008, you don’t have to wait for us any longer. In VS2008, VB added full support for many of the debugging features it lacked compared to C# in 2005. In particular we’ve added full support for the DebuggerDisplayAttribute.

By attributing a class or member with this attribute you can control how it is displayed in the debugger. For each column (name, value and type) you can provide an alternate string or expression to display.

The best part about this attribute is you can target types that exist in different libraries. You don’t even need the source for them. One of the members in the Type field which species the target type. Customizing a type in a separate library requires slightly more work than customizing a type you have the source for. For a source project you can just apply the attribute directly to the type or member and it will display. For a type in another library you need to do the following.

Define a class library and include all of the DebuggerDisplayAttribute you want. Make sure to apply the attributes to the assembly and specify the Type member. Ex.

<Assembly: DebuggerDisplay(“{ToString}”, Target:=GetType(Guid))>

Place the built library under the folder “Visual Studio 2008\Visualizers” which is under your my documents folder.

After doing this any Guid type will now show up as the actual Guid String (“10f3c4eb-7c0f-41b1-ae83-8838ff2f4f70”) instead of {System.Guid}

C Literal Number Grammar

2007-09-27T00:00:00+00:00

Recently I was investigating a bug and ended up investigating the possible values of C++ literals for char and number types. The following page has a very useful and detailed grammar of C/C++ literals that came in handy.

http://cpp.comsci.us/etymology/literals.html

Testing Your Code

2007-09-26T00:00:00+00:00

This is a little off topic from my typical purely technical discussion.

I’m a huge advocate of automated code verification. Unit testing is preferred but really any method is acceptable as long as it gets the job done of testing your code. IMHO, testing is the most important part of a developers job.

Besides the 3 or 4 projects I am doing as part of my job at Microsoft, I work on numerous side projects. Most of these come in the form of tools, libraries or little fun gadgets. These are written in a variety of languages; C#, VB, C++, Powershell Script, etc …

This is pretty common amongst many devs I know. We love our job and code before, during and after work. Some of the devs I know don’t bother automating testing of any of their tools. When asked they generally respond that it’s not worth it.

I disagree strongly for certain types of tools. For tools and libraries I own that I expect long term usage on, I test methodically. On any given day I may update a couple of the tools. The only item that adds sanity to the situation is testing. This allows me to respond quickly to customer requests for features, bug fixes, etc and get quick verification that I haven’t introduced a regression.

C Keyword Identifiers

2007-09-12T00:00:00+00:00

This falls into the “learn something new everyday but not necessarily entirely useful bucket”. An app I spend a bit of time on leverages the CodeDom heavily to spit out managed code. While running through some test cases the other day I noticed that it was prefixing many identifier names with the @ symbol when the generated code was C#.

At first I assumed this was a bug in my code but I noticed that the generated code compiled just fine. A bit of digging through the C# spec turned up the purpose of the @ character. This allows you to use C# keywords as identifiers. For instance you can have “class @class”.

http://msdn2.microsoft.com/en-us/library/aa664670(VS.71).aspx

VB has the same type of feature but they require the name to be wrapped in [].

This still didn’t answer the question of why my code generated with this character. It turns out this is a feature of the C# CodeDom. When outputting an identifier, the C# CodeDom will prefix the identifier with @ if one of the following conditions are true.

The identifier is a keyword
The identifier is prefixed with two underscores.

It turns out the second rule exists to allow flexibility in the C# compiler implementation. All identifiers that are prefixed with two underscores are inherently reserved for the implementation to provide such actions as extended keywords (reference at bottom of above link).

Singleton Pattern

2007-09-04T00:00:00+00:00

Quite awhile back I posted about how to create a re-usable singleton pattern in .Net. Link is here. A bit of time has passed and I’ve altered the pattern a bit. The reasons for the change are some new type inference patterns and FxCop cleanliness.

The first pattern I introduced had a couple of FxCop violations. Namely Microsoft.Design CA1000 - Do not declare static members on generic types. The logic here being that static methods don’t have any type inference benefits as you must explicitly add the type into the name of the type you were using (in this case Singleton).

Secondly because T was at a class level rather than a method level I couldn’t have granualar methods which had differing set of constraints. The result was a pattern that was not always easy to write out.

The new pattern takes care of both of these problems. It has two methods. One of which can be satisfied with a trivial lambda expression. The other can easily be used for classes that satisfy the generic constraint new() with no additional lambda.

class c1
{
    public static c1 Instance1 
    {
        get { return Singleton.GetInstance(() => new c1());}
    }

    public static c1 Instance2
    {
        get { return Singleton.GetInstance<c1>(); }
    }

    public c1()
    {

    }
    
}

Below is the new singleton pattern.

public delegate T Operation<T>();
/// <summary>
/// Used for classes that are single instances per appdomain
/// </summary>
public static class Singleton
{
    private static class Storage<T>
    {
        internal static T s_instance;
    }

    [SuppressMessage("Microsoft.Reliability", "CA2002")]
    public static T GetInstance<T>(Operation<T> op)
    {
        if (Storage<T>.s_instance == null)
        {
            lock (typeof(Storage<T>))
            {
                if (Storage<T>.s_instance == null)
                {
                    T temp = op();
                    System.Threading.Thread.MemoryBarrier();
                    Storage<T>.s_instance = temp;
                }
            }
        }
        return Storage<T>.s_instance;
    }

    public static T GetInstance<T>()
        where T : new()
    {
        return GetInstance(() => new T());
    }
}

#endregion
}

Edit: Originally forgot to add the signature for Operation

Debuggernonusercode And Properties

2007-08-29T00:00:00+00:00

DebuggerNonUserCode is an attribute that tells the debugger that the target item is not code typed by the user. It can be added to classes, structs, methods, constructors and properties.

The benefits of this attribute is that it allows the compiler and designers to distinguish user code from generated code. As such the debugging experience can be altered. When this attribute is present and “Just My Code” (JMC) is on the debugger will not step into or break in these methods for normal cases. Instead it will treat it is if it was a call to a framework assembly.

However if you type code like the following you won’t get the behavior you probably expect.

Class C1
    Private m_f1 As Integer

    <DebuggerNonUserCode()> _
    Property P1() As Integer
        Get
            Return m_f1
        End Get
        Set(ByVal value)
            m_f1 = value
        End Set
    End Property
End Class

While investigating a recent bug I found that I could step into the get/set method of properties annotated with DebuggerNonUserCode. The reason why is a bit unexpected. The attribute is applied to the property, not the get/set method. The debugger will only check the actual methods involved. It doesn’t special case properties in any fashion. If you want to get the expected behavior, you have to annotate the get/set method directly.

Class C1
    Private m_f1 As Integer

    Property P1() As Integer
        <DebuggerNonUserCode()> _
        Get
            Return m_f1
        End Get
        <DebuggerNonUserCode()> _
        Set(ByVal value As Integer)
            m_f1 = value
        End Set
    End Property
End Class

I’m not saying this is the best solution but it will work for both VS 2005 and VS 2008. IMHO ideally this attribute when applied to a property would affect both the get and set method.

C Frustration Unions And Constructors

2007-08-23T00:00:00+00:00

Using PreFast internally we recently came across a class of bugs. In several places we were using uninitialized structures as local variables. The error came about because the types in question had no default constructor. For instance take the following structure

struct s1
{
    int m1;
    int m2;
};

This structure does not have a default constructor and hence the values for a local are not guaranteed to be any value. At first I thought this would be a simple fix. All I needed to do was add a default constructor that initializes the members to known values. At first I wondered why the original type author hadn’t bothered to do this.

s1() : m1(0),m2(0) { }

Now this looks well and good, I re-compile and get a whole lot of errors. It turns out that we have several unions defined for s1.

union 
{
    s1 u1;
    int u2;
};

It’s not legal for a union member to have a non-trivial default constructor. The next solution is to use a C++ hack to initialize the variables on the stack.

s1 x = {0};

I consider this a hack for a few reasons

It only works if you need everything to be initialized to 0
It spreads out the logic for your type. Instead of being localized to the actual type definition code it’s now spread out to the type use.
It’s hard to maintain because if you ever add another member that requires different initialization you must update every single use of the type. Worst of all you can add said member and won’t get a single error or warning about it.

In the end for types where 0 didn’t make sense we ended up using a factory pattern to initialize the variables.

All I wanted was a constructor.

C Type Inference Change For Orcas

2007-08-07T00:00:00+00:00

While playing around with a batch of Orcas code recently I found a welcome change to the C# type inference rules for Orcas. The return type of a generic delegate can now be inferred from the actual return values. Here is some sample code demonstrating the problem.

class Program
{
    public delegate T Operation<T>();

    public static void M<T>(Operation<T> del)
    {

    }

    static void Main(string[] args)
    {
        M(delegate { return 42; });
}
}

In Whidbey the above code will fail with a compiler error.

error CS0411: The type arguments for method 'ConsoleApplication34.Program.M<T>(ConsoleApplication34.Program.Operation<T>)' cannot be inferred from the usage. Try specifying the type arguments explicitly.

This is really frustrating in Whidbey because the you have to modify the call to include the actual type specifier.

M<int>(delegate { return 42; });

However this code now works without modification in Orcas. This is a welcome and extremely useful change.

And yes, the same inference is possible with VB in Orcas. In Whidbey this was not an issue because VB did not support lambda expressions.

Module Module1

    Public Delegate Function Operation(Of T)() As T

    Public Sub M(Of T)(ByVal del As Operation(Of T))

    End Sub

    Sub Main()
        M(Function() 1)
    End Sub

End Module

Closures In Vb Part 6 Limitations

2007-08-06T00:00:00+00:00

For previous articles in this series please see …

As powerful as closures are in the language they do have a few limitations. We worked hard in Orcas to put as few limitations in Orcas as possible. Below are the current limitations and some insight into why they exist this way.

Cannot use “ByRef” parameters in a closure

Example:

Sub LiftAByRef(ByRef x As Integer)
    Dim f = Function() x
End Sub

Message: error BC36639: 'ByRef' parameter 'x' cannot be used in a lambda expression.

The problem here is the expectation surrounding x. Any change in the value of “x” inside the method “LiftAByRef” should be reflected in the calling function. Normally for any lifted parameter we add a new field inside the closure and all read/writes are redirected into that value. For “ByRef” parameters we would additionally have to ensure that all writes are make to the parameter. Even in the presence of an exception. Not a trivial task.

Cannot use “Me” in a closure created inside a structure.

Example:

Structure S1
    Public F1 As Integer

    Public Sub M1()
        Dim f = Function() F1
    End Sub
End Structure

Message: error BC36638: Instance members and 'Me' cannot be used within a lambda expression in structures

Closures capture values by reference. It’s not possible to capture the “Me” of a structure by reference in VB. The only other option is to capture them by value. If we did that then all changes to members of a structure inside a lambda would not affect the structure in which they were created; merely the value copy. This is very different behavior from every other place that closures are used. To avoid confusing behavior this is not a legal operation.

Cannot use a Restricted Type in a closure

Example:

Sub LiftRestrictedType()
    Dim x As ArgIterator = Nothing
    Dim f = Function() x.GetNextArgType().GetModuleHandle()
End Sub

Message: error BC36640: Instance of restricted type 'System.ArgIterator' cannot be used in a lambda expression.

This hopefully will not affect many users. There are several types in the CLR that are considered restricted because they have special semantics. Typically they are special cased by the CLR and as such we can’t use them in a closure. Several of these cannot be used in VB at all. They are …

System.TypedReference
System.ArgIterator
System.RuntimeArgumentHandle

Cannot Goto into scope that contains a closure

Example:

    Sub BadGoto()
        Dim x = 0

        GoTo Label1
        If x > 5 Then
Label1:
            Dim y = 5
            Dim f = Function() y
        End If
    End Sub

Message: error BC36597: 'GoTo Label1' is not valid because 'Label1' is inside a scope that defines a variable that is used in a lambda or query expression.

If you look back at Part 3 of this series you will see that a lot of work goes into initializing closures inside of a scope. Unfortunately allowing a user to jump into a block that contains a closure makes respecting these rules very difficult. In a even trivial example in makes the resulting code mostly unreadable. We decided to disable this in Orcas and reconsider it in a future release.

It is perfectly legal however to jump into any scope that is currently visible regardless of whether or not in contains a closure. Because jumping into a visible scope does not affect the creation of a variable lifetime (just the ending), it does not add any complications to the code.

Cannot mix “On Error Goto” and Closures

Message: error BC36595: Method cannot contain both an ‘On Error GoTo’ statement and a lambda or query expression.

Because of restriction #4 we must disable this scenario as well since it’s very easy to hit this scenario with “On Error Goto”.

Powershell Simple Tricks Number To Hex

2007-08-02T00:00:00+00:00

When investigating bugs a lot of times you start out with an unfamiliar HRESULT value. Searching the web will give you a hint of what the problem is and the real search can begin. If you get this error out of a log file it can come in a base 10 value versus a base 16 (hex). Searching for your base 10 code is much less efficient that the hex counter part. No matter.

C:\Users\jaredpar> "0x{0:x}" -f -2147023293 | out-clipboard

Expression Evaluator Sample Released

2007-08-01T00:00:00+00:00

The new version of the Visual Studio 2008 Beta2 SDK was just released. Among the many new samples is a basic expression evaluator sample. This sample shows you how to add basic debugger support to a managed language in Visual Studio.

This is the first version of the sample and mainly has support for locals and member expansion. I’m planning on adding more features in future releases. Since this is mainly to provide users with guidelines on how to build an EE, any requests we get will be given a lot of consideration.

http://www.microsoft.com/downloads/details.aspx?FamilyID=d9000e2c-bd3f-4717-a181-723960814e16&displaylang=en

Detecting If You Are An Admin

2007-08-01T00:00:00+00:00

This came up on an internal alias. A customer wanted to know how to determine if there were running as an admin in a tool. Below is a sample program that will print out whether or not you are the machine admin or a member of the machine administrators group.

This is essentially the same as the code in a previous post of mine but ported to VB.

Imports System.Security.Principal
Module Module1

    Function IsRunningAsLocalAdmin() As Boolean
        Dim cur As WindowsIdentity = WindowsIdentity.GetCurrent()
        For Each role As IdentityReference In cur.Groups
            If role.IsValidTargetType(GetType(SecurityIdentifier)) Then
                Dim sid As SecurityIdentifier = DirectCast(role.Translate(GetType(SecurityIdentifier)), SecurityIdentifier)
                If sid.IsWellKnown(WellKnownSidType.AccountAdministratorSid) OrElse sid.IsWellKnown(WellKnownSidType.BuiltinAdministratorsSid) Then
                    Return True
                End If

            End If
        Next

        Return False
    End Function

    Sub Main()
        Console.WriteLine("Is Admin {0}", IsRunningAsLocalAdmin())
    End Sub

End Module

Coding Quiz Anonymous Type Types

2007-08-01T00:00:00+00:00

Question: How can you create a variable in VB which is typed as an anonymous type but not actually create an instance of that type?

Answer in comments.

Note, doing this is not particularly useful it came about while I was playing around with a feature a few days ago.

Imetadataimport Getparamformethodindex

2007-07-30T00:00:00+00:00

While investigating a recent bug I found about an interesting return for GetParamForMethodIndex. On a perfectly verifiable assembly, a call to GetParamForMethodIndex was returning a failure code. After some searching I found the return code was CLDB_E_RECORD_NOTFOUND. I was surprised at first because it’s a verifiable assembly so how could the record for a parameter I knew existed not be there?

It turns out this is legal. GetParamForMethodIndex returns a mdParamDef token by which you can query for information about a parameter with GetParamProps. This will return the following information about a parameter.

Name of the parameter
Attributes about the param (ByRef, Marshal, etc …)
Default value

In this particular case the assembly was generated without a parameter name. As it also had none of the information the parameter row was omitted from the metadata. The reason is adding an empty row takes up space and provides no data.

Note you can’t reproduce this behavior with ILASM.exe. If you omit a parameter name, ILASM.exe will add one for you and hence generate a parameter row.

Closures In Vb Part 5 Looping

2007-07-26T00:00:00+00:00

For previous articles in the series please see

Once again sorry for the long delay between posts.

Looping structures can cause unintended consequences when used with Lambda expressions. The problem occurs because lambda expressions do not execute when they are constructed but rather when they are invoked. For example take the following code.

Sub LoopExample1()
    Dim list As New List(Of Func(Of Integer))
    For i = 0 To 3
        list.Add(Function() i)
    Next

    For Each cur In list
        Console.Write("{0} ", cur())
    Next
End Sub

Many users are surprised to find out the above will print “4 4 4 4 “. The reason goes back to my previous 2 posts on variable lifetime and scope. All “For” and “For Each” blocks in Vb have 2 scopes.

Scope where iteration variables are defined
Body of the for loop

The first scope is entered only once no matter how many times the loop is executed. The second is entered once per iteration of the loop. Any iteration variables that are defined in a For/For Each loop are created in the first scope (in this case “i” and “cur”). Hence there is only one of those variables for every loop iteration and the lambda function lifts the single variable “i”.

This has thrown off many users because the behavior works most of the time. For instance if I switched the code to run “Console.Write” inside the first loop, it would print out “0 1 2 3 “ as expected.

To mitigate against this problem the above code will actually produce a warning in VB.

warning BC42324: Using the iteration variable in a lambda expression may have unexpected results. Instead, create a local variable within the loop and assign it the value of the iteration variable.

There are two ways to fix this problem depending on the behavior you want. If you see this warning and don’t know if it affects you, the safest change is to do the following.

Sub LoopExample2()
    Dim list As New List(Of Func(Of Integer))
    For iTemp = 0 To 3
        Dim i = iTemp
        list.Add(Function() i)
    Next

    For Each cur In list
        Console.Write("{0} ", cur())
    Next
End Sub

This will cause “i” to be created in the second scope. Hence there will be a different value for every loop iteration and the code will print out “0 1 2 3” as expected.

If you do want the code to print out “4 4 4 4 “ then add “Dim i = 0” before the start of the loop.

Closures In Vb Part 4 Variable Lifetime

2007-06-15T00:00:00+00:00

For previous articles in this series please see

Sorry for the long delay between posts here. We’re getting Orcas out the door and getting this series completed takes a back door to shipping.

Originally I wanted to talk about looping structures next. However when I started writing that post I realized that I had to talk about lifetime before the looping structures would make sense.

Prior to Orcas the lifetime of a variable in VB was the entire function. This presented several problems from a closures perspective. Imagine you had a looping structure and the value was used in a lambda expression.

Sub LifetimeExample()
    Dim list As New List(Of Func(Of Integer))
    For i = 0 To 5

        Dim x = i * 2
        If True Then
            list.Add(Function() x)
        End If
    Next

    For Each f In list
        Console.Write(f() & " ")
    Next
End Sub

In this example if we left the lifetime rules unchanged, there would be a single variable “x” for the entire function. That means that we would end up printing out

10 10 10 10 10

This is somewhat unexpected and essentially means that VB could not support complex Lambda scenarios. To fix this we altered the lifetime of variables to be tied to the scope they were contained in. The end effect is that each iteration of the loop has a separate “x” since each iteration enters and leaves the scope of the “if” statement. As a result it will print out

0 2 4 6 8 10

We did make one backcompat adjustment for this change. The lifetime of variables in VB was visible if you tried to use an uninitialized variable in a loop/goto. For instance the following code will also print out 0 2 4 6 8 10 because it takes advantage of the fact that the variable “x” has a lifetime longer than the loop.

Sub VisibleLifetime()
    For i = 0 To 5
        Dim x As Integer
        Console.WriteLine(x)
        x += 2
    Next
End Sub

To make sure that we didn’t break any existing code we had one little errata for the change. When a variable’s scope is re-entered, and hence recreated, and it is not initialized to a value it will get the last value of the variable.

Defining Guids

2007-06-08T00:00:00+00:00

One of the tasks I find to be annoying during development is pasting GUID’s into my code. You need GUID’s for lots of different functions (COM, Constants, etc …). GuidGen is a good tool for getting the Guid into your code. However I don’t insert GUID’s very often. Everytime I do, I find I’ve forgotten where the tool is. Then I go through the process of locating it and getting my GUID out.

PowerShell to the rescue :)

PS> [Guid]::NewGuid().ToString() | out-clipboard

[Edit1] Corrected ToString->ToString() Tom Barnum’s feedback.

Expression Evaluator Sdk

2007-05-25T00:00:00+00:00

The Expression Evaluator is a VSIP component that allows Visual Studio Languages to plug in a Debugger. It’s what controls the information inside the locals,watch,auto, stack frame and immediate windows during a debugging session. Both VB and C# use this API to plug in their debugger displays to Visual Studio.

Here is a link the MSDN page overviewing this component.

http://msdn2.microsoft.com/en-us/library/bb161694(VS.80).aspx

The Expression Evaluator (EE) layer is very flexible and gives the library author a lot of room to create custom displays, evaluators and so forth. The only real downside to the layer is the documentation. Namely there isn’t much around :(

I use this layer as part of my daily dev life while working on the Visual Basic Debugger layer. After encouragement from a few customers and internal Microsoft employees I decided to start blogging about this VSIP layer. Mainly I want to help people who are writing Expression Evaluators out by giving tutorials on getting the fun things working (type visualizers, proxies, display attributes) and also warn about the various pitfalls involved with the API.

If you have any suggestions for entries or questions about the Expression Evaluator or Symbol Provider layer please post them in the comments or email me through the link. I’d really like this part of my blog to be customer driven given that there are a limited number of users who want to plug into this layer.

I’ll be tagging these entries wit the EESdk flag.

Closures In Vb Part 3 Scope

2007-05-25T00:00:00+00:00

For previous articles in this series please see

Thus far in the series we’ve only lifted variables that are declared in the same block/scope. What happens if we lift variables in different scope? The answer is that one closure class will be created for every unique scope where a lifted variable is declared and all of the variables in that scope that are lifted will be placed in that closure. Once again, examples speak best

Sub Scope1()
    Dim x = 5
    Dim f1 = Function(ByVal z As Integer) x + z
    Console.WriteLine(f1(5))
    If x > 2 Then
        Dim y = 6
        Dim g = 7
        Dim f2 = Function(ByVal z As Integer) z + y + g
        Console.WriteLine(f2(4))
    End If
End Sub

The code will end up looking like so …

Class Closure1
    Public x As Integer

    Function Lambda_f1(ByVal z As Integer)
        Return x + z
    End Function

End Class

Class Closure2
    Public y As Integer
    Public g As Integer

    Function Lambda_f2(ByVal z As Integer)
        Return y + z + g
    End Function
End Class

Sub Scope1()
    Dim c1 As New Closure1()
    c1.x = 5
    Console.WriteLine(c1.Lambda_f1(5))
    If c1.x > 2 Then
        Dim c2 As New Closure2()
        c2.y = 6
        c2.g = 7
        Console.WriteLine(c2.Lambda_f2(4))
    End If
End Sub

There are a couple of items to take away from this example.

Only two closure classes were created even though three variables were lifted. The number of closures only depends on the number of scopes of all of the lifted declared variables.
The closures are created at the begining of the scope they are associated and not at the begining of the method. This will be more important in the next part of the series.
Each lambda instance is attached to the closure associated with the scope the lambda is declared in.

The next twist is what were to happen if the lambda “f2” were to also use the variable “x”. As it’s currently written there is no association between Closure1 and Closure2 therefore there is no way for it to access the lifted variable. The answer is two fold. Firstly to reduce clutter I pasted the closure classes as if they were separate entries. In fact Closure2 would appear as a nested class of Closure1 in the real generated code.

Secondly if x were used inside of “f2”, the real use would be “c1.x”. That’s (almost) no different than “someOtherVar.x”. Therefore the instance of c1 will be lifted into Closure2.

Dim f2 = Function(ByVal z As Integer) z + y + g + x

Woud result in the following definition of Closure2 …

Class Closure2
    Public y As Integer
    Public g As Integer
    Public c1 As Closure1

    Function Lambda_f2(ByVal z As Integer)
        Return y + z + g + c1.x
    End Function
End Class

In deeply nested lambdas and scopes this type of lifting will continue recursively.

That’s it for this entry, the next article will talk about looping structures and possibly variable lifetime.

Closures In Vb Part 2 Method Calls

2007-05-03T00:00:00+00:00

For previous articles in this series, please see

Part 1: Introduction

This part of the series will focus on how method calls are handled in closures. As stated in the previous article, the purpose of closures is to allow all operations inside a lambda or query expression that would normally be available inside the function or sub. To do this closures often need to capture (or lift) relevant variables from the function into the generated class.

There are 2 types of methods and method calls that closures have to handle.

Method calls to a shared method or methods on modules.
Method calls to instance members of a class

**Scenario #1 **

Below is an example of a method call inside a lambda expression for scenario #1.

Module M1

    Function MyValue() As Integer
        Return 42
    End Function

    Sub Example1()
        Dim x = 5
        Dim f = Function() x + MyValue()
    End Sub

End Module

Here we are calling a module method inside a lambda. Module Methods or Shared methods can be called from anywhere because they require no specific variable for the call. This requires no special work from closures as the call can just be made naturally.

Class Closure
    Private x As Integer

    Function Lambda_f() As Integer
        Return x + M1.MyValue
    End Function
End Class

Scenario #2

Calling an instance method is more difficult than a shared method because it requires the referenc “Me”. If you don’t type this specifically in code the VB Compiler will add it for you under the hood. To make this work the closures code will also “lift” the variable “Me” in the same way that it lifts normal variables in a function.

Calling a instance method inside a lambda expression is little difference than calling a member method on a variable used in a lambda. The only difference is the variable is “Me”. For example

Class C1
    Private m_myValue As Integer

    Function MyValue() As Integer
        Return m_myValue
    End Function

    Sub Example2()
        Dim x = 5
        Dim f = Function() x + MyValue()
    End Sub
End Class

In this case we need to access both “x” and “Me.MyValue()” from the closure. The generated code will create space for both of these variables and the transformed code in Example2 will store both of the values.

    Class Closure
        Private x As Integer
        Private OriginalMe As C1
    
        Function Lambda_f()
            Return x + OriginalMe.MyValue()
        End Function
    End Class

    Sub Example2()
        Dim c As New Closure
        c.x = 5
        c.OriginalMe = Me
        Dim f = New Func(Of Integer)(AddressOf c.Lambda_f)
    End Sub

As usual, the generated code is much uglier but this essentially what will be generated. That wraps it up for method calls. In the next part, I will discuss the variable liftetime and scoping issues that come into play with closures.

Closures In Vb Part 1

2007-04-27T00:00:00+00:00

One of the features I implemented for VB 9.0 is lexical closure support. This a great addition to the VB language and I wanted to do a series of blog posts to describe this feature and how it will impact your code.

Lexical Closures (more often referred to as simply Closures) are the underpinnings for several new features in Visual Basic 9.0. The are part of the guts of Lambda and Query expressions. This will be a several part series on Closures in VB 9.0; how they work, their limitations, pitfalls surrounding their use.

To start off, let’s get a basic summary of what a Closure is. Wikipedia defines it as “… a is a semantic concept referring to a function paired with an environment …”. I prefer to describe it as follows. A closure is a feature which allows users to seemlessly access an environment (locals, parameters and methods) from more than one function. Even better are samples :)

Class C1
    Sub Test()
        Dim x = 5
        Dim f = Function(ByVal y As Integer) x + y
        Dim result = f(42)
    End Sub

End Class

In this code we have a lambda expression which takes in a single parameter and adds it with a local variable. Lambda expressions are implemented as functions in VB (and C#). So now we have two functions, “Test” and “f”, which are accessing a single local variable. This is where closures come into play. Closures are responsible for making the single variable “x” available to both functions in a process that is referred to as “lifting the variable”.

To do this the compiler will take essentially 4 actions.

Create a class which will contain “x” in order to share it among both functions. Call it “Closure” for now
It will create a new function for the lambda expression in the class “Closure”. Call it “f” for now
Create a new instance of the class “Closure” inside the sub “Test”
Rewrite all access of “x” into the member “x” of “Closure”.

Class Closure
    Public x As Integer

    Function f(ByVal y As Integer) As Integer
        Return x + y
    End Function
End Class

Class C1
    Sub Test()
        Dim c As New Closure()
        c.x = 5
        Dim f As Func(Of Integer, Integer) = AddressOf c.f
        Dim result = f(42)
    End Sub

End Class

Now “x” is shared amongst both functions and the user didn’t have to know anything about the code we generated. You can see from this simplified example just how much code Closures and all of the other new VB 9.0 features are saving you here (Type Inference, Lambda Expressions).

Note this is only a simulation of what is generated when you use a closure, the actual generated code is much uglier and involves lots of unbindable names “$Lambda_1”, etc …

In the next part of this article I’ll dive into some more uses of closures (multiple variables, method access, terminology, etc…).

Cosetproxyblanket And Managed Code

2007-04-19T00:00:00+00:00

When running FxCop on any managed code that uses CoSetProxyBlanket you will see an error message saying the method cannot be called reliably from managed code. I’ve hit that message before was frustrated by my attempts to find an explanation on the web. Part of the reason is I’m not the most efficient searcher on the web.

However and internal discussion revealed to me a great blog entry explaining exactly why this is unreliable and more importantly how to work around the issue.

http://blogs.msdn.com/mbend/archive/2007/04/18/cosetproxyblanket-not-supported-from-managed-code.aspx

Technorati tags: DotNet

Debugging Managed Code With Object Id S

2007-04-18T00:00:00+00:00

Just found out about a neat way to keep track of managed object references while debugging. Check out this blog entry

http://blogs.msdn.com/greggm/archive/2007/01/17/setting-conditional-breakpoints-using-object-ids.aspx

Readonly Textbox That Doesn T Look Funny

2007-02-12T00:00:00+00:00

When you make a WinForm TextBox ReadOnly, it aquires a distinctive look because it changes the background. Users often want the appearance of the TextBox to stay the same, they just don’t want it to be mutable. Here’s a snippet to make a TextBox ReadOnly and not change it’s appearance.

Dim saved As Color = Me.TextBox1.BackColor
Me.TextBox1.ReadOnly = True
Me.TextBox1.BackColor = saved

Blogging About Code

2007-02-09T00:00:00+00:00

It’s somewhat frustrating to try and post code to the internet for your blog. Getting the color scheme right isn’t easy if you’re like me and don’t use a standard color scheme for coding. I prefer a very dark coding environment (inkpot) and most people don’t want to read code in that scheme.

Someone just led me onto a great website that will auto-colorize your code.

http://puzzleware.net/codehtmler/default.aspx

Powershell And Svn

2007-02-01T00:00:00+00:00

I often delete versioned files from Visual Studio without great care for the source control impact. However I eventually have to go through and clean up the mess I made by not properly deleting the files from svn as well as on disk. When you delete a file physically but not in SVN, SVN willl declare the file missing.

For a couple of files a quick “svn delete foo.txt” will take care of the problem. When there are lots of files this is tedious and prone to error. That is, unless you have powershell.

# Used to do a "svn delete" on all missing files  
function SvnRemoveMissing()  
{  
    $data = & svn status  
    foreach ( $entry in $data )  
    {  
        if ( $entry -match "^!\s+(.*)$" )  
        {  
            & svn delete $($matches[1])  
        }  
    }  
}  

The Problem With Powershell Is That Sometimes You Can T Use It

2007-01-31T00:00:00+00:00

PowerShell puts the fun back in scripting and it’s horrfying but every now and again I’m forced to write a good old batch script. Batch is good enough to get most jobs done it’s just not as “fun” as PowerShell. Recently it came up in an internal alias about how to detect if you’re running on Vista from a batch script.

for /f "tokens=4 delims=.] " %%i in ('ver') do set OSVERSION=%%i

This parses the output of the “ver” command in batch and gives back the result. If %OSVERSION% is 6 then you’re on Vista.

Serialization Understanding The Problem

2007-01-29T00:00:00+00:00

Writing a proper serialization mechanism is ofter very difficult. The problem is most people don’t realize this because it just works in their application and .Net makes it very easy to do. A lot of the problem is not understanding what factors you need to consider when writing a serializer.

Failures with serialization can be classified in two ways

Critical - Deserialization is impossible. Typically resulting in a runtime exception or fata error
Incorrect - Deserialization succeeds but the underlying data is incorrect in some fashion

For this post I use the term “box” describes a place where all the factors which can interfere with proper serialization differ. Below is a list of all of the factors I know of that can affect serialization. And by affect I mean that serializing data in one “box” and deserializing it in another “box” could have an affect on the data.

Process - There are types of data which are only valid within the process where they are created. For instance when you open a file in Windows, you are given a handle. The handle is only valid for the process where the file was opened, passing it to another process by value will cause an error in the target process. Or worse the handle will be valid in the target process but will not point to the same file.
Computers - This problem is a superset of the process problem. In addition to the problems of a process you have to deal with setup. For instance binary serialization in .Net takes an inherit dependency on assemblies at a specific version¹. If the target computer does not have these assemblies then deserialization will fail.
TimeZone / Where - If you embed time in your data as a string and serialize/deserialize in different time zones you will encounter an “Incorrect” deserialization failure.
Thread - There are particular types of data that are only valid within a specific thread (thread local storage, COM pointers). Depending on the type of data it can cause varying failures in an application. One good example of this are COM pointers. COM has strict marshalling rules with respect to threads². Passing a COM interface pointer between threads without calling into appropriate API (CoMarshalInterThreadInterfaceInStream , CoGetInterfaceAndReleaseStream).
Application Domain - Application Domains in .Net are essentially mini-processes. While it’s probably valid to pass a file handle between application domains, it’s not always valid to pass managed protected resources between two domains. Every application domain in a .Net process can have different security permissions and hence it’s possible to pass references to in-accessible resources to other application domains.
BigEndian / Little Endian - If you’re doing custom serialization to bytes you also need to be concerned with the endian-ness of the machine you’re working on. It’s possible (albeit hard) to get into this problem and most people won’t ever have to deal with it for a managed application. I don’t know of a good way to hit this in managed code but it’s definately a problem for cross platform application and unmanaged code.

Most of the problems I’ve seen with respect to serialization is because people embed a “box” sensitive resource in their serialization data but don’t realize it. When desigining serialization for your data you need to consider all of the factors above which affect your application.

The good news is most applications don’t have to consider all of the above factors into their application. Below is a breakdown of some common category applications and the list of factors they have to consider.

MyApp Serializers - The data is serialized for my application and my application only. It’s a hobby app or a small office app. These are the easiest because they usually only have to take into account factor #1. They don’t care about application domains, computer->computer communication and the like. They have some data they’d merely like to preserve between sessions in a file somewhere for this particular user.
Product Serializer - The is a for sale global product and reliability is key. But once again the data is only serialized by and for my application. They are a bit harder because they are expected to work in more situations. They may have to consider a wider range of scenarios including threads and application domains.
Cooperative App - This app serializes data for use in other “boxes”. These are the hardest to write because they often have to

There are ways to relax these dependencies but at the end of the day some version of the assembly must be on the machine. ↩
Lookup COM Apartments for more information. ↩

Publishing Web Application Projects Via Ftp

2007-01-26T00:00:00+00:00

Web Application projects are a new project type in Visual Studio 2005 SP1. It almost all of the niceties of the web projects with the semantics of being contained within a class library project. I like the feel of them and do all of my web app development in them now.

I just ran into a small problem publishing them. After banging my head on the wall for an hour I thought to post the solution in case anyone else ran into the problem.

One of the data files I’m using is rather large so I opted to publlish via FTP for speed purpose. After uploading the app I couldn’t view any of my pages. The error page kept telling me there was an error becaus the page “/Application.Master” was not present on the system. These apps all ran fine on my local system.

As you can guess, it’s not to easy to turn up useful information searching the web when your key words are “master page file parse error”.

The problem is that because I only ever published this application via FTP, front page extensions were never engaged and hence the project was not created as far as the ASP.Net system was concerned on the server. Instead it was just another directory. I did a mini publish via HTTP and my app immediately started working.

Boolean Parameters

2007-01-23T00:00:00+00:00

An item I try to avoid in any API I create are methods which …

Take more than one parameter
One of the parameters is a boolean

Typically this pattern indicates a True/False option on an operation type. This boolean value turns on or off some type of scenario for that operation.

The reason I dislike this is that it’s not maintainable. I constantly forget what the boolean value means and it makes my code less readable to people who are reviewing. The reviewer sees SomeOperation(1, true) and they constantly ask “What does true mean?”. And even though most programmers won’t admit this, years after you write the code and you’re tracking down a bug, you also forget what true meant in this case.

Instead I prefer to break down the operation into three different methods. Say the operation in question loaded a particular piece of data from a file. The data block can have references to other data blocks. In some cases I only want the first layer of data and in others I would like to load all referencable data blocks. Here is the break down of operations I will create.

public LoadDataBlock(identifier)
public LoadDataBlockRecursive(identifier)
private LoadDataBlockImpl(identifier, recursive)

The private implementation takes the boolean indicating the option. This gives the implementor of the method the flexibility of having the boolean option parameter. The big win here is for the caller. They don’t need to know about this hidden parameter and instead will have two clear API method calls.

Determining If You Re An Administrator From Powershell

2007-01-19T00:00:00+00:00

Here’s a handy PowerShell function I used to determine if I’m currently running as an Administrator in PowerShell

# Determine if I am running as an Admin  
function AmIAdmin()  
{  
 $ident = [Security.Principal.WindowsIdentity]::GetCurrent()  
  
 foreach ( $groupIdent in $ident.Groups )  
 {  
  if ( $groupIdent.IsValidTargetType([Security.Principal.SecurityIdentifier])
)  
  {  
   $groupSid = $groupIdent.Translate([Security.Principal.SecurityIdentifier])  
   if ( $groupSid.IsWellKnown("AccountAdministratorSid") -or
$groupSid.IsWellKnown("BuiltinAdministratorsSid"))  
   {  
    return $true;  
   }  
  }  
 }  
  
 return $false;  
}  

Where Is This Script

2007-01-18T00:00:00+00:00

Scripts often need to dynamically find out what directory they are executed from. In CMD scripts this is done by %~dp0. For powershell the following will do the trick.

split-path -parent $MyInvocation.MyCommand.Definition

When this is run anywhere in the script body (not a function within the script) it will return the directory currently holding the running script. If you execution this from a function within a script though, it will print out the body of the function. If you need to know the directory of the script inside the body of the script, store it in script level variable.

$script:mypath = $MyInvocation.MyCommand.Definition
function printScriptPath() { return $script:mypath; }

Passing Data Between Asp Net Pages

2007-01-16T00:00:00+00:00

When developing an ASP.Net page I tend to pass a lot of data between pages. A lot of it comes from being fairly OO natured and wanting to have a page that displays a particular type of content.

There are lots of articles detailing how to pass data between pages that have a 1-1 relationship using the PreviousPage property and PreviousPageType directive. That is great for wizard style pages where there is only one page that is allowed to pass data to a particular page. However it’s not as helpful when the target page is meant to take input from multiple sources because you can only specify a single PreviousPageType directive.

The approach I take is interface based. The PreviousPage property is available whether or not the PreviousPageType directive is present in the page. All the PreviousPageType directive does is make the property strongly typed. Without it the directive it’s just typed as Page. To make it strongly typed, I create an interface that the calling page must implement and then attempt to cast the source page into the specific interface.

Partial Public Class _Default
  Inherits System.Web.UI.Page

  Protected Sub Page_Load(ByVal sender As Object, ByVal e As System.EventArgs) Handles Me.Load
    Dim source As ITextSource = TryCast(Me.PreviousPage, ITextSource)

    If source IsNot Nothing Then
      Display(source) 
    Else
      ...
    End If
  End Sub
End Class

Public Interface ITextSource
  ReadOnly Property Text() As String
End Interface

This gives us all of the benefits of using the PreviousPageType directive with just a few lines of overhead. It makes passing data between pages trivial.

Windows Forms Event Lifecycle

2007-01-08T00:00:00+00:00

When deveploping windows forms app, it’s important to understand the event lifecycle of a form. That way you know what code to put where to ensure it’s loaded at the appropriate time. That being said I wrote a small app to detail the events in a basic windows form application. Greater indentations indicate nested events

Form Startup

OnHandleCreated
OnCreateControl
1. OnLoad
OnActivated
OnShown

Form Shutdown

OnClosing
OnClosed
OnDeactivate
OnHandleDestroyed

C Preprocessor And Template Identifiers

2006-12-19T00:00:00+00:00

A couple of hours of tracking down a compiler error a couple of days ago taught me something about the preprocessor I’d like to pass on so others can avoid the … learning :).

I kept getting confronted by “constant expression must be followed by a ,”. Not the exact error but in the right range. This was a bit annoying because the error occurred on a simple template. Something along the lines of the following

template<typename T1, typename T2> 
struct Foo
{
  ...
}

The long and short of it is the C/C++ processor will replace any word in a file if there is a corresponding macro with the same name. This I knew from years of programming. Every now and again you learn something new though. The preprocessor will also try and expand template identifier’s. Normally this isn’t a problem if the macro is just defined to another simple value. For example, the below usually won’t cause you a headache

#define KEY KEY2

template <class KEY>
class Pair
{
  ...
};

However if you try something fancy like the following, bad things start to happen

#define KEY (KEY+2)

T1 was not the actual template identifier but it was a rather obscure word. Some other header file being include was defining it to some complex macro. At the end of the day, I switched the template definition header file to be included first and to thus avoid the macro problem. I also added a few #define/#error combinations so that an error would occur if they were included in the wrong order

Dispose Vs Delete

2006-12-14T00:00:00+00:00

Programs allocate resources for use during execution. The problem with resources is that they are limited and often times need to be recycled. Languages devise constructs and patterns for developers to periodically free up resources so that their programs can continue running.

In Native C++ you typically dealt with two kinds of resources that needed recyling.

Memory
Operating System Resources (usually handles of one form or another)

C++ uses the “delete” operator and destructors to handle reclying both resources. When “delete” is called on an object it will call it’s destructor and then free the memory (in that order). Operating system resources are typically freed in the destructor of a C++ object (Resource Aquisition is Ini tialization). So in one fell swoop the “delete” operator handles recycling both C++ types of resources.

.Net is garbage collected and hence only has to deal with one kind of resource.

Operating System Resources

.Net has no “delete” call because you’re not meant to be freeing memory in yourself (it’s the CLR’s job). .Net objects which handle operating system resources typically free them up via the Dispose interface. Most languages (including C# and VB) provide a “using” pattern by which you can easily hold onto a resource for a given time and guarantee it will be freed at the end of the operation.

The problem comes when people familiar with C++ “delete” semantics attempt to apply them to the .Net dispose pattern. Dispose does not free managed objects memory. The memory of the object is still available and can be accessed just fine.

This is not saying that’s a good idea (in fact it’s typically a bad one). Most objects provide for very unpredictable behavior once Dispose has been called and hence you shouldn’t be using them.

Lua Msi Luanch Program Now On Vista

2006-12-06T00:00:00+00:00

I’m a huge fan of the LUA support in Vista. It has it’s quirks but it’s a major step forward for Windows programming. As a former *nix guy I’ve had to run LUA the hard way before Vista. The support in Vista is tons better than it used to be.

There are a few areas that are still not optimal. One of them that’s been bugging me lately are legacy MSI installers. Vista almost always correctly elevates and offers to run the installer as Admin.

This causes a small problem though with some legacy MSI’s. A lot of MSI have a check box at the end of the MSI setup with text similar to “Launch SuchAndSuch Program Now”. The problem is who the program launches as.

Prior to Vista there was no auto-priviledge elevation. So most people ran MSI’s as there normal user account which was an admin. Launching the process at the end of the MSI was just fine.

Now in Vista most people run as a normal user and MSI’s elevated. So this feature will now, on legacy MSI’s, launch the program as an Administrator rather than your normal user account.

Without the help of a tool, there’s no good way to detect when this happens. To be safe, I always un check that box and manually start the app so I know it will be running under my normal account.

Using Powershell To Update Acls

2006-11-29T00:00:00+00:00

I’ve been an avid fan of running as a limited user account (LUA) for almost all of my computing career. It’s in large part a holdover from my *NIX days but it’s an important practice for Windows as well. I’m very excited about this being the default for Windows going forward starting with Vista.

The bad news is that not everyone is quite LUA compliant yet. This hits me hardest with PowerShell when I’m trying to run some auto update scripts that change the Registry. There are a lot of programs I use that still store their settings in HKLM or the likes. So I’ve added a PowerShell script that will grant me access to those keys as an LUA. It was written a long time ago so it doesn’t utilize the get/set-acl commandlets but it runs just the same.

# Give the user access to the registry entry
function GrantLuaRegistryKey([string]$regkeyPath)
{
	[string]$ident = "{0}\{1}" -f ${env:\userdomain}, ${env:\username};
	$rights = [Enum]::Parse([Security.AccessControl.RegistryRights], "FullControl"); 
	$allow = [Enum]::Parse([Security.AccessControl.AccessControlType], "Allow");
	$iFlags = [Enum]::Parse([Security.AccessControl.InheritanceFlags], "ContainerInherit,ObjectInherit");
	$pFlags = [Enum]::Parse([Security.AccessControl.PropagationFlags], "None");
	$rule = new-object Security.AccessControl.RegistryAccessRule $ident,$rights,$iFlags,$pFlags,$allow;
	$key = [Microsoft.Win32.Registry]::LocalMachine.OpenSubKey($regkeyPath, $true);
	
	# Add the rule to the collection
	$col = $key.GetAccessControl();
	$col.AddAccessRule($rule);
	$key.SetAccessControl($col);
	$key.Close();
}

Combobox Selecteditem Selectedvalue Selectedwhat

2006-11-07T00:00:00+00:00

ComboBox has a lot of helpful properties that allow you to get access to items selected by the Control. The problem is knowing which one is best in which scenario. The documentation is not 100% clear on what the values will be in various situtations. The most important factor to know is whether or not you are DataBinding in the ComboBox.

DataBinding Case

In this case the properties will have the following values

SelectedItem - For gets this will return the actual object in the DataSource that is being displayed in the ComboBox. For sets if the value exists in the DataSource, it will be selected, otherwise the operation will complete without an exception but won’t actually do anything.
SelectedValue - This property depends on the value of ValueMember.

If the property ValueMember is not Nothing the ComboBox will look for a member on SelectedItem with the name specified in ValueMember and return that. This is also the value displayed in the ComboBox.

If the property ValueMember is Nothing, then the SelectedValue will return .ToString() on the SelectedItem

SelectedIndex - Index of SelectedItem in the DataSource

Non-DataBinding Case

SelectedItem - Gets and Sets both go to the currently selected object from the Items collection
SelectedValue - Will be Nothing/null
SelectedIndex - Index in the Items collection of the SelectedItem

Using Powershell To Automate Redundant Tasks

2006-10-23T00:00:00+00:00

It’s nice to have powershell when you need to do a lot of redundant file work. I’m use Subversion as the source code control backing for a lot of my hobby projects. I have pretty much all the code I’ve ever written as a hobby or for school stored in SVN.

The biggest downside to using Subversion is integration tracking. Subversion doesn’t do a lot of in house integration tracking (it just looks like more edits). If you follow the subversion guidelines this isn’t much of a problem and everything will go as expected.

Unfortunately I didn’t follow those guidelines. I glossed over one of the most important parts and as a result I lost track of the integrations I did and didn’t do. So today when I tried to do a large merge, SVN got confused and saw tons of my files as having conflicts. When SVN hits a conflict during any operation it will mark the file as “conflict” put all of the changes into the main file (Call it foo.cs) and then create the left and right files (foo.cs.merge-left.rxxx, foo.cs.merge-right.rxxx). Left is typically the file on your disk with the right file being the one you’re trying to merge in.

Really I just wanted the changes from the most recent revision (the right file in this case). In this case since I just wanted the newest version the process was to copy the right file and mark the file as resolved. Lots of tedious hand work.

Powershell let me fix the problem with 1 minute of script work.

I still think one the most valuable uses for PowerShell is just playing with .Net. I knew that I could use a regex to match the files with right files and capture the actual name of the file as well.

It took only two quick command line experiments with PowerShell to determine the way of grouping the name of the file as part of the regex. Result being that if you separate a regex into groups with parens then they will be added (by number) into the $matches variable. So the following will match all *.merge-right.rXXX files and capture the original file name in $matches[1].

"(.*)\\.merge-right.*"

So I ran the following script in my directory that was having issues and all was well again.

PSH:> foreach ( $file in dir) { if ( $file.Name -matches "(.*)\\.merge-right.*" ) { copy $file.Name $matches[1]; svn resolved $matches[1] } }

Responding To Windows Shutdown

2006-10-20T00:00:00+00:00

.Net 2.0 Added a lot in the way of allowing programmers to easily interact with Windows. One of the best additions is the Microsoft.Win32.SystemEvents class. It holds events for a lot of system relatied events (hence the name).

A common scenario I see on the forums is users wanting to respond to a system shutown event. The easiest way to access this from managed code is the Microsoft.Win32.SystemEvents.SessionEnding event. This is not exactly the shutdown event because it actually represents the user session that the program is running in. A session is created when you log into the system and is destroyed when you log off or shutdown the computer (which eventually forces a logoff :) ).

The arguments provide two pieces of useful information. The first being the reason for the session ending (logoff or shutdown). It also allows you to cancel the event via the Cancel property. For example the following code will prevent the shutdown of the computer as long as the program is running.

Imports Microsoft.Win32

Sub PreventShutdown()
  AddHandler SystemEvents.SessionEnding AddressOf PreventShutdownHelper
End Sub

Sub PreventShutdownHelper(ByVal sender As Object, ByVal args As SessionEndingEventArgs)
  If args.Reason = SessionEndReasons.Shutdown Then
    args.Cancel = True
  End If
End Sub

While this will prevent the shutdown from occurring you will see many of the apps on your screen close. That’s because some of them get the shutdown event before your app and most will close as a result.

There are a couple of caveats that come with using this event.

If you’re using a WinForm app, there is no gaurantee you will get this event before your application raises the Close event.
This event won’t ever fire (without a lot of trickery) from a Console application.
Setting Cancel to True will usually work but not always.

That aside, if you just need to save some data and or do some quick book keeping before exiting this will do the trick.

Custom Combobox

2006-10-13T00:00:00+00:00

This is another forum request. I’ve seen multiple requests from users who are looking to customize the drop down window for the ComboBox control. There is no builtin way to do this with the existing control. Instead you must create a custom solution.

Start by adding a new UserControl to your windows application or library project. Have it inherit from ComboBox and call it CustomComboBox.

This ComboBox will display a user provided control for the drop down window. We will be using an instance of System.Windows.Form (named m_form) to replace the drop down window. The idea is to have the control to essentially float below the ComboBox much like the drop down window does today. To do this we need to custom the Form to not show up in the task bar and to not have a titlebar or default buttons.

We also need to prevent the ComboBox from droping down the builtin window. There is no way to actually stop the ComboBox from dropping down the window. Instead we need to make it appear that the drop down didn’t display by setting it’s height to 1 pixel.

To make the logic a bit easier, we will also provide a default empty control to be displayed when the drop down is expanded

Here is the full text of the constructor used to initialize the CustomComboBox

Public Sub New()

        ' This call is required by the Windows Form Designer.
        InitializeComponent()

        ' Setup the form to display the control
        m_form = New Form()
        m_form.StartPosition = FormStartPosition.Manual
        m_form.FormBorderStyle = FormBorderStyle.None
        m_form.Hide()
        m_form.ShowInTaskbar = False

        Me.Control = New Control()   ' Default Control
        m_dropDownHeight = Me.DropDownHeight
        Me.DropDownHeight = 1       ' Prevent the DropDown from showing
    End Sub

The next step is to have the form display in the proper place when the drop down button is hit. To do this we override the OnDropDown() method and add the display logic there.

Protected Overrides Sub OnDropDown(ByVal e As System.EventArgs)
    MyBase.OnDropDown(e)

    If Not m_form.Visible Then
        DisplayControl()
    End If

    Me.DroppedDown = False
End Sub

Private Sub DisplayControl()
    Dim loc As Point = Me.PointToScreen(Point.Empty)
    loc.Y += Me.Height

    m_form.Location = loc
    m_form.Show()
End Sub

The last item left is to add a property for the user to set the custom control to display. To be flexible we will allow instances of Control. It will be docked in the form. We also need to listen to the LostFocus event on the control. This is the key for us to hide the form like when the user clicks away from the normal ComboBox.

Public Property Control() As Control
    Get
        Return m_control
    End Get
    Set(ByVal value As Control)
        If Not m_control Is Nothing Then
            m_form.Controls.Remove(m_control)
            RemoveHandler m_control.LostFocus, AddressOf Me.OnControlLostFocus
        End If

        m_control = value
        m_control.Dock = DockStyle.Fill
        AddHandler m_control.LostFocus, AddressOf Me.OnControlLostFocus
        m_form.Controls.Add(m_control)
    End Set
End Property

Private Sub OnControlLostFocus(ByVal sender As Object, ByVal e As EventArgs)
    m_form.Hide()
End Sub

Powershell Tutorial

2006-10-12T00:00:00+00:00

PowerShell is an exciting product coming out of Microsoft. It’s a shell technology many old *nix junkies will find refreshing and very intuitive. It’s based on .Net and has a syntax feels like a mix between Pearl, C# and VB. I’ve blogged about it before but due to several extraneous factors I stopped using it for awhile.

I’ve recently started using it again while trying to reintruduce myself I ran accross the following tutorial. It appears to be written on a version of PowerShell shortly before it was renamed from it’s code name Monad. The syntax is slightly off in a place or two but it’s a great introducion to powershell none-the-less.

http://arstechnica.com/guides/other/msh.ars/2

742971

2006-09-06T00:00:00+00:00

A highly rated post on the VB MSDN Forum recently asked “How can I open up a web browser when a user clicks on a button?”

The first problem is “How can you start a web browser to a specified URL”. This is actually very easy to do in windows. You only need to create a process where the name is the URL you are trying to navigate to and windows will take care of the rest. This is quickly demonstrated by clicking on Start->Run and typing in an url (http://www.msn.com).

Doing it from Code is almost as simple. Make sure that System.Diagonstics is imported into your project and the following Sub will work.

Public Sub OpenUri(ByVal target As Uri)
    Dim info As New ProcessStartInfo()
    info.FileName = target.ToString()
    info.UseShellExecute = True
    Process.Start(info)
End Sub

Second problem is how to make a button open the link when it’s clicked. For this you’ll just need to handle the Clicked event and call the above Sub with the appropriate URL.

But why stop there? The UI Web Controls have a LinkButton class that does much the same but there is no such class in System.Windows.Forms. Lets create one.

First step is to create a new Windows Control Library project. Then add a new User Control called LinkButton. Open up the Designer file and make it derive from Button.

Now we need to add a field to allow the developer to specify the link to open. This field should be of type URI to make sure we’re passed a valid URI. When setting the Link if there is currently no text displayed we should just display the link for ease of use.

        Private m_link As Uri
    
        Public Property Link() As Uri
            Get
                Return m_link
            End Get
            Set(ByVal value As Uri)
                m_link = value
    
                If m_link IsNot Nothing Then
                    If String.IsNullOrEmpty(Me.Text) Then
                        Me.Text = m_link.ToString()
                    End If
                End If
            End Set
        End Property

Now we need to actually open the link when the user clicks on the button. Override the OnClick method and put in essentially the same code as above.

Protected Overrides Sub OnClick(ByVal e As System.EventArgs)
    MyBase.OnClick(e)

    If m_link Is Nothing Then
        MessageBox.Show(Me, "Must specify a valid Uri to display", "Error", MessageBoxButtons.OK, MessageBoxIcon.Error)
        Return
    End If

    ' Start a process to open the link in a browser
    Dim info As New ProcessStartInfo()
    info.FileName = m_link.ToString()
    info.UseShellExecute = True
    Process.Start(info)
End Sub

While our LinkButton works at this point, it looks just like any other button. To make it more apparent that this is a LinkButton we will have the text be underlined and blue so it looks more like a hyperlink

Public Sub New()

    ' This call is required by the Windows Form Designer.
    InitializeComponent()

    ' Add any initialization after the InitializeComponent() call.
    Me.Font = New Font(Me.Font, FontStyle.Underline)
    Me.ForeColor = Color.Blue
    Me.Text = String.Empty
End Sub

At this point we have a fully functional LinkButton class. Lets make it a bit better though. It’s possible for the developer to have the text of the button be completely different than the actual Link. For instance the button could say “Hello” and the link could point to www.msn.com. We’ll add a tooltip so the user can see where the link actually goes to.

Open up LinkButton in the designer and drag a tooltip onto the surface. Rename it m_toolTip. Now lets update the code that sets the link

Public Property Link() As Uri
    Get
        Return m_link
    End Get
    Set(ByVal value As Uri)
        m_link = value

        If m_link IsNot Nothing Then
            If String.IsNullOrEmpty(Me.Text) Then
                Me.Text = m_link.ToString()
            End If
            m_toolTip.SetToolTip(Me, m_link.ToString())
        Else
            m_toolTip.SetToolTip(Me, "No Link Provided")
        End If
    End Set
End Property

At this point we have a fully functional link button that is ready for re-use. See the attachment for the full code sample.

715339

2006-08-23T00:00:00+00:00

A frequent question I see on the forums is how can you drag controls around in a form with the mouse. Drag and drop is a very big and complex subject but this is one subset that can be implemented very quickly in VB. For this sample I’ve created a simple form with Visual Studio and added a TextBox named TextBox1. This sample will work for virtually any control though with a little modification.

To implement this fetaure we need to handle 3 events on the TextBox

MouseDown
MouseUp
MouseMove

We need to handle the Up and Down events so we can record whether or not the user clicked on the TextBox. As opposed to the user clicking elsewhere on the form (or anywhere else on the screen for that matter). In addition to knowing the user clicked on the TextBox we need to also record where the Mouse was when the user Clicked.

This brings up the issue of screen versus client coordinates. Events like MouseMove have a Location property which contains a Point relative to the client. In Client Coordinates 0,0 referes to the top left corner of the control. In Screen Coordinates 0,0 refers to the top left corner of your actual monitor. To do a move we to find the difference in the MousePoints. It’s easier to do this using Screen Coordinates.

Whenever we detect a mouse move event, we need to calculate how much the mouse moved in terms of x and y. Then modify the location of the control by the same amount and the control will be dragable.

Public Class Form1

    Private m_mouseHeldOnTextBox As Boolean
    Private m_mouseHeldPoint As Point

    Private Sub OnTextBoxMouseMove(ByVal sender As System.Object, ByVal e As System.Windows.Forms.MouseEventArgs) Handles TextBox1.MouseMove
        If m_mouseHeldOnTextBox Then
            Dim currentPoint As Point = TextBox1.PointToScreen(e.Location)
            Dim xDelta As Integer = currentPoint.X - m_mouseHeldPoint.X
            Dim yDelta As Integer = currentPoint.Y - m_mouseHeldPoint.Y
            TextBox1.Location = New Point(TextBox1.Location.X + xDelta, TextBox1.Location.Y + yDelta)
            m_mouseHeldPoint = currentPoint
        End If
    End Sub

    Private Sub OnTextBoxMouseDown(ByVal sender As System.Object, ByVal e As System.Windows.Forms.MouseEventArgs) Handles TextBox1.MouseDown
        If (e.Button = Windows.Forms.MouseButtons.Left) Then
            m_mouseHeldOnTextBox = True
            m_mouseHeldPoint = TextBox1.PointToScreen(e.Location)
        End If
    End Sub

    Private Sub OnTextBoxMouseUp(ByVal sender As System.Object, ByVal e As System.Windows.Forms.MouseEventArgs) Handles TextBox1.MouseUp
        m_mouseHeldOnTextBox = False
    End Sub
End Class

665101

2006-07-13T00:00:00+00:00

Several people have asked on the newsgroups about a good method to bind a list of items to entries in a ToolStripDropDown instance. I’ve run into this issue a couple of times and wanted to share some sample code.

The class takes in a ToolStripDropDown instance. You can then set the data source to any IEnumerable collection. Whenever the data in the list changes, call ReloadFromSource(). Or alternately, if the list implements IBindingList then it will automatically update.

Imports System.ComponentModel

Public Class DropDownBinding
    Private m_dd As ToolStripDropDown
    Private m_list As IEnumerable
    Private m_bindingList As IBindingList

    Public Sub New(ByVal dd As ToolStripDropDown)
        m_dd = dd
        m_list = New List(Of Object)()
    End Sub

    Public Sub SetDataSource(ByVal e As IEnumerable)
        If Not m_bindingList Is Nothing Then
            RemoveHandler m_bindingList.ListChanged, New ListChangedEventHandler(AddressOf Me.OnListChanged)
        End If

        m_bindingList = TryCast(e, IBindingList)
        m_list = e

        If Not m_bindingList Is Nothing Then
            AddHandler m_bindingList.ListChanged, New ListChangedEventHandler(AddressOf Me.OnListChanged)
        End If

        RebuildList()
    End Sub

    Public Sub ReloadFromSource()
        RebuildList()
    End Sub

    Private Sub OnListChanged(ByVal sender As Object, ByVal e As ListChangedEventArgs)
        RebuildList()
    End Sub

    Private Sub RebuildList()
        m_dd.Items.Clear()

        For Each cur As Object In m_list
            m_dd.Items.Add(cur.ToString())
        Next
    End Sub
End Class

663898

2006-07-12T00:00:00+00:00

After working a little over 2 years with the VSTS Architect SKU I’ve decide to take on a new role at Microsoft. I am taking a job with the Visual Basic Core Compiler team. This is an exciting oppuritunity for me to get back to a lot of the topics I worked with in college. Expect me to be showing up with more regularity on VB newsgroups and forums.

The transition occurred just over a month ago but I’ve been too busy to put an announcement on my blog until now.

579204

2006-04-19T00:00:00+00:00

While I was looking into CAS changes in 2.0 I found some very good news about thread changes in 2.0. In .NET 2.0 both Thread.CurrentPrincipal and the Thread Impersonation token are now propagated amongst threads including ThreadPool.QueueUserWorkItem. The latter was a great pain to developers in .NET 1.0 particularly because Thread.CurrentPrincipal did not propagate and it created a lot of funny code to deal with the CAS impacts.

This MSDN article delves into these changes. http://msdn.microsoft.com/msdnmag/issues/04/11/NETMatters/

Follow Up Pinvoke With 32 And 64 Bit

2006-04-13T00:00:00+00:00

Recently after I made my original post titled ‘PInvoke with 32 and 64 bit’, a developer emailed me. They pointed out that there is an internal structure that dynamically resizes based on the architecture; IntPtr. IntPtr will be size of the native chip pointer value. So for values like size_t that map to the pointer size, you can use the Address field of IntPtr.

This approach has some small down sides.

The type you are marshaling must have the same size as IntPtr for all of the architecture’s you run on
If you expose an interop API that uses IntPtr in this manner, you have to find a good way to inform your users that they really need to pass the value in the address of the IntPtr

A way to get around #2 is to define your own type (like I did with SizeT) and just wrap an IntPtr. If you have no other fields in your type it will Marshall like an IntPtr. You can provide friendly properties that assign values into the IntPtr’s Address property.

If you have a type who’s size does not change in the same way as IntPtr then you’ll probably have to go the custom marshal route

575011

2006-04-12T00:00:00+00:00

One of the big difficulties still remaining in .NET 2.0 is the interoping with native API’s which takes in types that have different sizes on 32 and 64 bit machines. For most API’s this is not a problem. But for many, including most of the memory API’s for example, the signature is technically different from 32 to 64 bit with respect to parameter size.

The problem occurs if any of the API’s parameters have a different size on 32 and 64 bit. IMHO, most common Win32 API parameters are composed of types that do not have different sizes on 64 bit (DWORD, UINT, LONG, etc ‘). The biggest culprit for me has been size_t.

Unfortunately there is no predefined solution for these types in .NET 2.0. So one has to be devised. The biggest issue is that you typically run the same .NET binary or 32 and 64 bit. So we need a type whose size is determined at runtime based on hints from the platform. Defining a type like this just isn’t possible with .NET. The best recourse is to define a fixed size type and a custom marshaler that will make the runtime size decision.

As a proof of concept I added a type SizeT to my interop library and a custom marshaling class to do the dirty work for 32 and 64 bit platforms. Here are the definitions. `

 public sealed class SizeT  
 {  
  private ulong m_value;

  public ulong Value  
  {  
   get { return m_value; }  
   set { m_value = value; }  
  }

  public SizeT(ulong val)  
  {  
   m_value = val;  
  }

  [SuppressMessage("Microsoft.Usage", "CA2225")]  
  public static implicit operator SizeT(ulong value)  
  {  
   return new SizeT(value);  
  }

  [SuppressMessage("Microsoft.Usage", "CA2225")]  
  public static implicit operator ulong(SizeT value)  
  {  
   return value.Value;  
  }  
 }

 public sealed class SizeTMarshaler : ICustomMarshaler  
 {  
  [SuppressMessage("Microsoft.Usage", "CA1801", Justification="data parameter
is a hidden requirement of the API")]  
  public static ICustomMarshaler GetInstance(string data)  
  {  
   return new SizeTMarshaler();  
  }

  #region ICustomMarshaler Members

  public void CleanUpManagedData(object ManagedObj)  
  {  
   // Nothing to do  
  }

  public void CleanUpNativeData(IntPtr pNativeData)  
  {  
   Marshal.FreeCoTaskMem(pNativeData);  
  }

  public int GetNativeDataSize()  
  {  
   return IntPtr.Size;  
  }

  public IntPtr MarshalManagedToNative(object ManagedObj)  
  {  
   SizeT value = (SizeT)ManagedObj;

   checked {

   IntPtr ptr = Marshal.AllocCoTaskMem(IntPtr.Size);  
   if (IntPtr.Size == 4)  
   {  
    Marshal.StructureToPtr((uint)value.Value, ptr, true);  
   }  
   else if (IntPtr.Size == 8)  
   {  
    Marshal.StructureToPtr(value.Value, ptr, true);  
   }  
   else  
   {  
    throw new ArgumentException("Invalid Pointer Size");  
   }

   }

   return ptr;  
  }

  public object MarshalNativeToManaged(IntPtr pNativeData)  
  {  
   if (IntPtr.Size == 4)  
   {  
    uint val = (uint)Marshal.PtrToStructure(pNativeData, typeof(uint));  
    return new SizeT(val);  
   }  
   else if (IntPtr.Size == 8)  
   {  
    ulong val = (ulong)Marshal.PtrToStructure(pNativeData, typeof(ulong));  
    return new SizeT(val);  
   }  
   else  
   {  
    throw new ArgumentException("Invalid Pointer Size");  
   }  
  }

  #endregion  
 }

There are two ways to determine if your interop API’s are affected by the move to a 64 bit platform.

Brute Force. Find all of the native types that you use and determine if there are different on 64 bit
Turn on FxCop Portability rules and it will catch the majority of your errors.

573690

2006-04-11T00:00:00+00:00

Just ran accross this site. It’s very helpful if you deploy any ClickOnce application

http://www.windowsforms.net/FAQs/default.aspx?PageID=1&CategoryID=24&tabindex=2

573476

2006-04-11T00:00:00+00:00

This is my first attempt at cross-posting to multiple blogs. Sorry for the spam.

573414

2006-04-11T00:00:00+00:00

Yet another reason to enable FxCop is it double checks some interop defines (Portability and Interop groups). After enabling FxCop on one of my home brewed interop libraries I was surprised that it complained I had some 64 bit portability problems with my HeapCreate and other Heap* API’s.

This API in particular caught me off guard because I did a 64 bit portability check of my interop library recently I specifically looked up that function. I verified that Heap* used DWORD’s for sizes which compile to 32 bit numbers on 64 bit.

After the FxCop warning I took yet another look and sure enough HeapCreate is defined to take in DWORD’s.

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/wcekernl/html/_wcesdk_Win32_HeapCreate.asp

However in my experience FxCop is generally correct if a bit paranoid in places. After confirming it caught slip ups in a different interop API I decided to triple check my Heap* definitions. When in doubt go to the source (winbase.h in this case). Sure enough FxCop is correct and msdn is wrong. Doing a more refined search I found that the correct documentation is available on windowssdk.

http://windowssdk.msdn.microsoft.com/library/default.asp?url=/library/en-us/memory/base/heapcreate.asp

I consider this further proof that FxCop is a great tool that should be used for any library development (home and most especially professionally).

In retro-spec if you take a very close look at the first link you’ll notice it’s actually the API for WinCE. I find it strange that it turns up higher in search results that the normal documentation.

572533

2006-04-10T00:00:00+00:00

I’ve started experimenting with Code Access Security at home. This derives from my desire to better understand .NET security and my recent fascination with ClickOnce applications.

This weekend I learned a hard lesson in security. I have a couple of ClickOnce apps. ClickOnce Visual Studio integration has a great feature that will root around in your code and attempt to determine the minimal permission set needed to run your application.

I started playing with this on one of my relatively simple applications. It contains several forms and opens up a couple of WebRequest objects but nothing terribly special. However the analysis said that my app needed to run with Full Trust (highest level). Unfortunately the app did not provide any more information besides that.

It took an hour of rooting around but in the end I discovered the culprit was the PropertyGrid instance on my form. It turns out that the PropertyGrid requires “Full Trust” to be used even though it’s not noted in the documentation.

489942

2005-11-07T00:00:00+00:00

One of the LUA tasks I found very frustrating was detecting whether or not I was running as an Admin in a .BAT script. It’s very difficult to do this correctly and the best way I found was to write a separate program that would determine this for me. This has a couple of limitation, the main one being that it adds extra dependencies to your scripts and makes them harder to deploy.

Monad has greatly simplified this task as I can do the detection code uses simple scripts.

# Determine if I am running as an Admin  
function AmIAdmin()  
{  
 $local:ident = [System.Security.Principal.WindowsIdentity]::GetCurrent()  
  
 foreach ( $local:groupIdent in $ident.Groups )  
 {  
  if (
$groupIdent.IsValidTargetType([System.Security.Principal.SecurityIdentifier])
)  
  {  
   $local:groupSid =
$groupIdent.Translate([System.Security.Principal.SecurityIdentifier])  
   if ( $groupSid.IsWellKnown("AccountAdministratorSid") -or
$groupSid.IsWellKnown("BuiltinAdministratorsSid"))  
   {  
    return $true;  
   }  
  }  
 }  
  
 return $false;  
}

I’ve added this to my profile so now it’s really easy to determine if I can complete an action or need to prompt for credentials in my scrips.

489114

2005-11-04T00:00:00+00:00

Determine if a command exists in your path. Very helpful when you are writing scripts for mulptiple machines.

$local:cmd = get-command doesnotexist -ea SilentlyContinue
if ( $null -eq $cmd )
{
  # Action if command does not exist
}
else
{
  # Action if command exists
}

This will get the directory holding the currently executing script

$local:scriptPath = [System.IO.Path]::GetDirectoryName($MyInvocation.MyCommand.Definition)

487904

2005-11-01T00:00:00+00:00

I spent a bit of my college days in *nix land. Much of that time was spent writing and maintaining scripts for my various shell accounts. After I switched to primarily developing on Windows, I was frustrated by the Windows CMD shell and eventually lost interest in scripting.

Lately I’ve been playing with Monad at work and it’s revived my interest. It takes a few minutes to get used to the new noun-verb command structure but once you do it’s extremely powerful and fun to use. I am looking forward to writing some commandlets once I have a few spare cycles. Until then just interesting scripts.

ps | sort-object ProcessName | where { ($_.HandleCount -gt 0) } | select-object ProcessName,HandleCount | Format-table -autosize

481096

2005-10-14T00:00:00+00:00

COM interop is one of the messy places where error by return value vs exception meet head on. COM was built on the basis of HRESULT return values while .NET tends to use exceptions. To compensate for that, the CLR helps out by mapping failure HRESULTs into .NET exceptions during COM Interop. It does this by the following process

Discard the HRESULT return value
Convert any [out, retval] components to the method return value

This is helpful in a lot of cases. You can make you’re COM calls happily and exceptions will be thrown when an error occurs. There are a lot of cases where this isn’t enough information for the developer. The main one are the semi-successful HRESULT return values. The CLR will only throw exceptions for failed HRESULTS ¹ but will not for values such as S_FALSE.

Values such as S_FALSE hold a lot in certain COM components and cannot be ignored in many situations. But because the CLR disregards this value there is no way to get to that value via this method. To be able to access the HRESULT in COM component return values you have to turn off the exception mapping. Unfortunately you will not be able to use the type library importer for this.

Modify your Managed Interafec to match the COM signature (add the int,uint return value and readd the [out,retval] component to the paramaters).
Decorate the method with the [PreserveSig] attribute.

This will allow you to access the HRESULT for the return value good or bad. The downside is that the CLR will no longer translate bad HRESULTS into exceptions and your back to using SUCCEEDED and FAILED methods.

List of Mapped HRESULTs to .NET Exceptions

http://msdn2.microsoft.com/en-us/library/9ztbc5s1

The documentation on MSDN says it will throw for anything but S_OK. You can verify that isn’t correct by using a component that returns S_FALSE

“This posting is provided “AS IS” with no warranties, and confers no rights” ↩

462471

2005-09-08T00:00:00+00:00

It’s important in multi threaded programs to understand the difference between an assigment operation and Interlocked.Exchange when you are dealing with shared references (memory).

The .NET ECMA spec in section 12.6.6 sets that reference assigment (and any other platform word size set) will be an atomic operation (when memory is properly aligned). I’ve seen some devs sight this when using simple assigment operations on a shared reference. The problem is the assigment is atomic but it makes no gaurantees about when the update will reach other threads (i.e. memory locations, caches). So it will cause no apparent errors but the change may not be immediately visible to other threads

Interlocked.Exchange while atomic also gaurantees that the change will be propagated to all other threads in the program. It’s a bit slower though.

458076

2005-08-30T00:00:00+00:00

It’s very important to read the documentation of this method before actually using it. A common misunderstanding of this method is that it will throw if the path does not exist. If Path.GetFullPath() discovers that the directory does exists it will make sure the user has access to the path. In the abscence of a directory it will skip the check and just return the canonicalized path.

This produces some interesting behavior with relative paths. Say that your program exists in c:\foo\MyApp.exe and you run it from that directory. If you called Path.GetFullPath() and passed in “..\..\..\..\..\bar.cs”, what would you expect it to do? My initial guess was that it would throw since that path goes above the root directory however it will just pretend it didn’t and return c:\bar.cs.

446830

2005-08-02T00:00:00+00:00

http://www.proudlyserving.com/archives/2005/08/dos_aint_done_t.html

Adam Barr debunks the “Dos aint done until Lotus 1-2-3 won’t run” myth. Includes links to the recent ones that sufaced about IE as well.

443159

2005-07-25T00:00:00+00:00

While working on a recent bug a question came up about how permissions work on NTFS. Take the following example directory path

c:\University\Fall2005\Homework.cpp

Now imagine that our user University\john had the access rights that were specified below. For completeness assume full control for Local System and Administrators.

University: Read & Execute, List Folder Contents, Read (All Allow)
Fall2005: Read & Execute, List Folder Contents, Read (All Deny)
Homework.cpp: Full Control

The question we had was whether or not University\John would be able to successfull access (read, edit, save) Homework.cpp. Some of us with Unix backgrounds guessed that he would not while most of the rest said that he would.

After a couple of minutes of discussion we cooked up an example and it turns out that University\John does indeed have Full Control of the Homework.cpp file. To be honest, this caught me completely by surprise. I did quite a bit of *nix programming in college and was incorrectly relying on my knowledge of the *nix file system particulars to answer this question.

If this were a similar *nix example (say /University/Fall2005/Homework.cpp) our user john would not have access to the file. When a file lookup is done in Linux, the VFS layer starts at the base path and looks up every file system object from the start of the path to the actual file. If the current process does not have sufficient permissions to access any object on the path, the search fails (as it would for the Fall2005 object).

I don’t know a lot about NTFS but I can only assume that it performs a lookup in a different manner. Then again, assumption got me into this problem …

441079

2005-07-20T00:00:00+00:00

http://liihs.irit.fr/dragice/foldndrop/

Check out the video on this website. The author has implemented a new way of dragging and dropping files in Java and has videotaped a demo. Interesting stuff

Marshaling Nested Data Structures Part 4

2005-07-18T00:00:00+00:00

This is part 4 of a series. You can find part one here. Please refer to that article for all of the Native definitions of the structures that I use here.

In the previous article we were left with a solution where using the code was very clean but the actualy implementation had extra allocation and perf overhead. We’ll conquer both of those in this installment by implementing a custom marshaler for our Course object. This is accomplished by implementing ICustomMarshaler and call the class CourseMarshaler. (ICustomMarshaler docs)

Custom marshaling is exactly like it sounds. The marshaler itself does practically nothing. Your code must transform the managed data structure into a native format and vice versa. It can be tedious at times but this method can be used to marshal even the most complex of structures. There are 6 methods you must implement. Lets go over each of them.

CleanupData() - Use this method to dispose of anything on your managed object when it’s no longer needed. Typically there is nothing to do here and this method remains blank as it does in our case.

CleanupNativeData() - Use this method to free up any data associated with the Native pointer after the runtime is finished using it. In our case, we have to allocate memory to Marshal the data structure into, so we need to free up this pointer.

public void CleanUpNativeData(IntPtr pNativeData)
{
    Marshal.FreeCoTaskMem(pNativeData);
}

GetNativeDataSize() - Returns the size of the unmanaged data structure. As previously discussed, this size is 268.

MarshalManagedToNative() - Takes in an object and returns a pointer to the memory that contains the Native structure in memory. To complete this we need to allocate a block of memory, and then marshal each of the fields in order into that memory block and return the pointer to the front of the block. The pointer we allocate will be freed later when the runtime passes it into CleanupNativeData(). Some error checking was removed for the sake of brevity.

public IntPtr MarshalManagedToNative(object managedObj)
{
    Course course = (Course)managedObj;

    IntPtr ptr = Marshal.AllocCoTaskMem(this.GetNativeDataSize());
    if (IntPtr.Zero == ptr)
    {
        throw new Exception("Could not allocate memory");
    }

    // Write the Int values in order into memory
    Marshal.WriteInt32(ptr, 0, course.Id);
    Marshal.WriteInt32(ptr, Marshal.SizeOf(typeof(Int32)), course.Count);
            
    // Now we need to Marshal each of the Student elements into the "array".  This 
    // starts immediately after the Ints
    IntPtr cur = new IntPtr(ptr.ToInt32() + (2 * Marshal.SizeOf(typeof(Int32))));
    for (int i = 0; i < course.Count; i++)
    {
         Student student = course.Students[i];
         Marshal.StructureToPtr(student, cur, false);
         cur = new IntPtr(cur.ToInt32() + Marshal.SizeOf(typeof(Student)));
    }
    return ptr;
}

MarshalManagedToNative() - Marshal the Native Struct into a managed version. This is almost identical to the sample that we did in part 3. Code reposted below.

public object MarshalNativeToManaged(IntPtr ptr)
{
    int courseId = Marshal.ReadInt32(ptr);
    int count = Marshal.ReadInt32(ptr, Marshal.SizeOf(typeof(Int32)));

    // Set the int values
    Course course = new Course();
    course.Id = courseId;
    course.Count = count;

    // Now read out the Student structures
    ptr = new IntPtr(ptr.ToInt32() + (2 * Marshal.SizeOf(typeof(Int32))));
    for (int i = 0; i < count; i++)
    {
        Student student = (Student)Marshal.PtrToStructure(ptr, typeof(Student));
        course.Students[i] = student;
        ptr = new IntPtr(ptr.ToInt32() + Marshal.SizeOf(typeof(Student)));
    }

    return course;
}

GetInstance() - This method is not a part of the ICustomMarshal interface but it’s a static method that must be implemented by any custom marashaler. The runtime uses this to create an instance of the object.

Now we have a complete implementation of ICustomMarshal. Really the only new method that we had to implement was MarshalManagedToNative() and that’s just the opposite of what we did in part 3. There are a couple of tidbits left that we have to alter.

The first is that we must convert the managed Course from a struct to a class. This is very important. A custom marshaler can only be applied to reference types. Changing Course to a class has a couple of other implications as well. Structs in C# are stored in the stack and reference types are stored on the heap. This has implications to Marshalling as well. When you Marshal a struct (or any ValueType), the runtime is expecting to find a stack based value (or better, a non pointer value) on the Native end. Now we are Marshalling a reference type so the runtime will expect to find a pointer value on the other end.

Also we can do away with the StructLayout attribute on the Course type. We are hand Marshaling this now so we don’t need to provide any hints to the runtime. Now we are left with just a vanilla class.

public class Course
{
    public int Id;  
    public int Count;  
    public List<Student> Students = new List<Student>(5);
}

Now we just need to inform the runtime about how to link our custom marshaler (called CourseMarshaler in my code) to the Course class. Every place that we declare a P/Invoke method we need to add custom Marshalling data. This is done by adding the MarshalAs attribute. Lets use the GetCourseInfo() method for an example. Here is our updated definition.

[DllImport("Enrollment.dll", CharSet = CharSet.Unicode)]
[return: MarshalAs(UnmanagedType.CustomMarshaler, MarshalTypeRef=typeof(CourseMarshaler))]
public static extern Course GetCourseInfo(int id);

We’ve made two important changes here as well. The first is we changed the return type of the method from an IntPtr to a Course. Remember that this is a class now so the runtime is expecting a pointer value. We take advantage of that here. Also we’ve added the MarshalAs attribute to the return type to tell the runtime to use the Custom marshaler that we created.

This makes the use code even cleaner since we aren’t dealing with an IntPtr return type anymore.

static void Main(string[] args)
{
    Course course = Enrollment.GetCourseInfo(42);
    Student first = course.Students[0];
}

That essentially concludes this series on Marshalling Nested Data Structures. I may add an additional chapter on common tips for debugging common marshalling problems if I have some time. Hope you enjoyed this.

This posting is provided “AS IS” with no warranties, and confers no rights. Use of included script samples are subject to the terms specified at http://www.microsoft.com/info/cpyright.htm.

440072

2005-07-18T00:00:00+00:00

Don’t forget to join us this Wednesday at 10am PST to discuss the new Team System Testing infrastructure. Topics include the Profiler, Test Tools (Unit, Generic, Manual), Web Load Testing, and Code Analysis (FxCop & PREFast). We have questions for you, will answer questions from you, and will chat about the exciting new technology. We’ll be in the “MSDN Chat Room”.

http://msdn.microsoft.com/chats

Marshaling Nested Data Structures Part 3

2005-07-14T00:00:00+00:00

This is part 3 of a series. You can find part one here. Please refer to that article for all of the Native definitions of the structures that I use here.

In part 2 of this series I demonstrated how you can Marshal nested data structures by flattening an array of structures into it’s individual elements. Using this is very awkward and is only practical when the size of your array is small. This installment will get around these limitations with a bit of Marshalling help. We’ll get back indexing and remove the awkward flattening.

Once again, don’t forget that the only 2 things that matter when Marshalling data

Byte Size
Byte Layout

When an array of structures are declared as a member of a struct, in the same manner as the Course structure, the data is inlined after the first two ints. We can get this data to Marshall properly by inserting a structure with the same size as the array of Students. The Native Student Structure is 52 bytes so we need a 260 byte structure. The following will do.

[StructLayout(LayoutKind.Sequential, CharSet = CharSet.Unicode, Size = 260 )]
public struct ArrayBlob
{
}

[StructLayout(LayoutKind.Sequential, CharSet = CharSet.Unicode)]
public struct Course
{
    public int Id;
    public int Count;
    public ArrayBlob Blob;

    /* ... Methods ... */
}

This structure meets all of our requirments. Course is now properly formatted and can be Marshalled back and forth successfully. However our student array is stuck in a chunk of data that has no access methods. This blob is just an array of students in memory. Getting the Students out in C would be a snap. We could just grab the address, cast it the the appropriate pointer and run away with the Student elements.

Student *elements = (Student*)(&(myCourse.Blob); 

Unfortunately I’m giving these examples in Safe code so this is not allowed. You can get the same effect with C# although it’s a bit slower and forces a memory allocation. We implement this as in indexer into the Course struct.

public Student this[int index]
{
    get
    {
        if ( index >= Count )
        {
            throw new ArgumentOutOfRangeException();
        }

        IntPtr ptr = Marshal.AllocHGlobal(Marshal.SizeOf(typeof(ArrayBlob)));
        try
        {
            Marshal.StructureToPtr(this.Blob, ptr, false);
            int offset = index * Marshal.SizeOf(typeof(Student));
            IntPtr structPtr = new IntPtr(ptr.ToInt32() + offset);
            return (Student)Marshal.PtrToStructure(structPtr, typeof(Student));
        }
        finally
        {
            Marshal.FreeHGlobal(ptr);
        }
    }
}

Some error checking was removed for brevity sake. Granted this is still a suboptimal way of accessing the members of the structure. Every index requires an allocation. But if you only interop for brief times and in non perf sensitive areas this code will do just fine. Here’s an example.

public static class Enrollment
{
    [DllImport("Enrollment.dll", CharSet = CharSet.Unicode)]
    public static extern IntPtr CreateStudent(
        [MarshalAs(UnmanagedType.LPWStr)] String firstName,
        [MarshalAs(UnmanagedType.LPWStr)] String lastName,
        int bDay,
        int bMon,
        int bYear);

    [DllImport("Enrollment.dll", CharSet = CharSet.Unicode)]
    public static extern void UpdateStudentInfo(
        [MarshalAs(UnmanagedType.LPWStr)] String firstName,
        [MarshalAs(UnmanagedType.LPWStr)] String lastName,
        int bDay,
        int bMon,
        int bYear,
        ref Student student);

    [DllImport("Enrollment.dll", CharSet = CharSet.Unicode)]
    public static extern IntPtr GetCourseInfo(int id);
}


class MarshalFun
{
    static void Main(string[] args)
    {
        IntPtr ptr = Enrollment.GetCourseInfo(42);
        Course course = (Course)Marshal.PtrToStructure(ptr, typeof(Course));

        Student first = course[0];
        Student second = course[1];
    }
}

This code assumes that at least 2 students were returned from GetCourseInfo(). This code is much cleaner than the previous example. It will also work with very large sized arrays. However it does have undue perf and memory overhead. Next time we’ll look at a way to have the clean code and remove all of the perf problems present in the current code.

This posting is provided “AS IS” with no warranties, and confers no rights. Use of included script samples are subject to the terms specified at http://www.microsoft.com/info/cpyright.htm.

Marshaling Nested Data Structures Part 2

2005-07-12T00:00:00+00:00

This is part 2 of a series. You can find part one here. Please refer to that article for all of the Native definitions of the structures that I use here.

The most important thing to remember when Marshalling data is that the .Net Runtime really only cares about byte layout and how many bytes you are trying to move. If you mangle your managed structure to match the byte size and layout of your unmanaged structure, the runtime will Marshal it sucessfully almost all of the time. Note by successfully I mean that the method will complete without causing the dreaded AccessViolationException but the data might not be quite what you were expecting.

For example. Here is the typical managed definition for the Student struct definition.

[StructLayout(LayoutKind.Sequential, CharSet = CharSet.Unicode)]
public struct Student
{
    [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 10)]
    public string FirstName;

    [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 10)]
    public string LastName;

    public int BirthDay;
    public int BirthMonth;
    public int BirthYear;
}

Notice that the size of this structure is 52 bytes.

FirstName 2 (size of char) * 10 (size of array)
LastName 2 (size of char) * 10 (size of array)
12 bytes (4 bytes for each of the int’s)

This is the same size as the native definiton of Student. In memory these bytes will all appear in order with no spaces between them (FirstName and LastName are set as ByValTStr’s so they will be inlined). So LastName is 20 bytes offset of a student structure, BirthDay is 40 bytes offset and so on.

You don’t always have to be so perfect though. In this case it’s only important because we want to accesss the fields of the managed structure. If this was not important to us, we could have just as easily declared the structure as so.

[StructLayout(LayoutKind.Sequential, Size = 52, CharSet = CharSet.Unicode)]
public struct Student
{ 

}

Note that i didn’t leave anything out in the structure above. It has the correct CharSet and Size so this could be successfully used wherever we need the Student structure. It just doesn’t help us very much in Managed code :)

Now onto the Course. A lot of developers get tripped up when trying to define structs like Course in managed code. Don’t forget, all we need to do is get the bytes correct and the Marshaller will work out the rest.

This structure is very small. There are only 5 inlined elements in the student array. This means that we could easily define our structure as so.

[StructLayout(LayoutKind.Sequential, CharSet = CharSet.Unicode)]
public struct Course
{
    public int Id;
    public int Count;
    Student Student0;
    Student Student1;
    Student Student2;
    Student Student3;
    Student Student4;
}

I realize there are several downsides to defining our structure like this. For one, it doesn’t look very clean. Secondly we’ve lost the ability to easily index the array of students. Also this works easily because there are only 5 elements in the array. If there were much more than that this wouldn’t be pheasible at all. However this structure is layed out exactly like the one defined in Native code so this will Marshal correctly.

The next part to this series will try to make things a bit more pleasing to the eye. We’ll get back indexing at the cost of a bit of manual Marshalling.

This posting is provided “AS IS” with no warranties, and confers no rights. Use of included script samples are subject to the terms specified at http://www.microsoft.com/info/cpyright.htm.

Marshaling Nested Data Structures Part 1

2005-07-11T00:00:00+00:00

A frequent question that pops up on newsgroups such as microsoft.dotnet.framework.interop is how to Marshal nested structures (and arrays of nested structures) via P/Invoke. The documentation on the subject that I have found usually only covers structures that defined in the UnmanagedType enumeration and just touch on the idea of structures containing nested arrays of custom structures. Part of the reason is that the explinations for these problems is quite involed and scenario specific. After answering it a few times I decided to make a multi part blog post on the subject. This will consist of at least 4 entries (might add some more if something comes up).

Index

The first installment is just going to be laying out our Native data structures that will be used throughout this series. I’m going to use a generic(and quite contrived) university classroom examle. Here are the C struct definitions and a couple of functions that I will be using.

struct Student 
{ 
  WCHAR FirstName[10]; 
  WCHAR LastName[10]; 
  int BirthDay; 
  int BirthMonth; 
  int BirthYear; 
}; 

struct Course{ 
  int Id; 
  int Count; 
  Student Students[5]; 
}; 

    // Create a student with the specified information.  Returned structure should be freed with 
    // GlobalFree
__declspec(dllexport) Student* CreateStudent(
    LPCWSTR firstName, 
    LPCWSTR lastName, 
    int bDay, 
    int bMon, 
    int bYear)

    // Update an existing student with the specified information
__declspec(dllexport) void UpdateStudentInfo(
    LPCWSTR firstName, 
    LPCWSTR lastName, 
    int bDay, 
    int bMon, 
    int bYear, 
    Student *student)

    // Get information for a particurlar course.  Returned structure should be freed with 
    // GlobalFree
__declspec(dllexport) Course *GetCourseInfo(int courseId)

The environment I am coding and testing my fixes in is Microsoft Visual Studio 2005 Beta 2. All of these examples should work in all versions of Visual Studio though. If you find an inconsistencies please let me know. All native examples will also be in Unicode.

This posting is provided “AS IS” with no warranties, and confers no rights. Use of included script samples are subject to the terms specified at http://www.microsoft.com/info/cpyright.htm.

436501

2005-07-07T00:00:00+00:00

Valery Pryamikov has released an update version of his RunAsAdmin tool. This version has a lot of new features including policy and stronger integration with the shell. This version is still in Beta but it’s been very stable for me.

http://www.harper.no/valery/PermaLink,guid,9dfd759b-6d53-4b97-8db8-350b32d68ab5.aspx

433489

2005-06-28T00:00:00+00:00

For the last few weeks, I’ve been playing around with a LUA tool that was recently presented to me. The tool is available on the following website

http://www.harper.no/valery/CategoryView,category,RunAsAdmin.aspx

The tool intercepts interactive logons and uses the Safer APIs to restrict your logon token to that of a normal user and start explorer with the restricted token. In effect, when you log on it logs you on as a normal user. The handy part is that it also creates a tray application that allows you to start programs with your full token rights without the hassle of entering your password. This is very similar to Aaron Margosis’s MakeMeAdmin.cmd script. I highly recomend this for anyone who runs as an LUA.

Here are the few downsides I have encountered while using this program

CTRL+SHIFT+ESC starts TaskMgr with full administrative rights (noted on website). You can get around this by choosing the taskmgr from the TaskBar right click menu
For this to work, your account must be a part of the Administrators group. Thus if you use the MakeMeAdmin script out of habbit one day, it will remove your account from the Administrator group.

423000

2005-05-29T00:00:00+00:00

A week ago, I blogged about how to register COM objects as LUA. This is a follow up post, to show you how to alter your standard ATL registration code so that your DLL will register for the current user instead of for the machine.

This code uses the RegOverridePreDefKey() method to override HKCR with the registration point for the current user (HKCU\Software\Classes).

    // DllRegisterServer - Adds entries to the system registry  
    STDAPI DllRegisterServer(void)  
    {  
        // registers object, typelib and all interfaces in typelib for the current
        user  
        HKEY key;  
        if ( ERROR_SUCCESS != ::RegOpenKeyW(HKEY_CURRENT_USER, L"Software\\\Classes",
        &key) )  
        {  
            return E_FAIL;  
        }  
        if ( ERROR_SUCCESS != ::RegOverridePredefKey(HKEY_CLASSES_ROOT, key) )  
        {  
            ::RegCloseKey(key);  
            return E_FAIL;  
        }  
        HRESULT hr = _AtlModule.RegisterServer(true);  
        ::RegCloseKey(key);  
        return hr;  
    }

“This posting is provided “AS IS” with no warranties, and confers no rights”

420187

2005-05-19T00:00:00+00:00

Ran accross this paper that does a good job of examining the history of content distrbution. Couple of years old now so it leaves out such items as Bittorrent but still a good read.

http://www.bearcave.com/misl/misl_tech/msdrm/darknet.htm

418780

2005-05-17T00:00:00+00:00

Believe it or not, you can develop, register and use COM objects with an LUA account ¹. COM objects are typically registered at one of one of two places.

HKLM\Software\Classes
HKCU\Software\Classes

Some people think that COM objects are registered under HKEY_CLASSES_ROOT (HKCR). This is partially true. HKCR is not a true registry key in the same way as HKEY_CURRENT_USER (HKCU) and HKEY_LOCAL_MACHINE (HKLM). Instead it is a hybrid key that merges the two keys above giving precedence to the information stored in HKCU. So whenever you open HKCR you are really opening a comibined view of the machine default COM settings and the user COM settings. This combined view will give precedence to user settings. This means that when you create an object with CoCreateInstance() or similar APIs, COM will use an object defined in HKCU, before HKLM.

But wait! ATL generated scripts all write their information to HKCR, how does that work? When you create a new key or value under HKCR, it will default to creating this key under HKLM. Unless the key already exists under HKCU. In that case the information will be stored in its existing location under HKCU. This means that you can develop, register and use COM applications all as an LUA. You can update most RGS files it ATL projects and force them to register under HKCU\Software\Classes instead of HKCR.

The is really only one small gotcha to this but it typically only affects services. HKCR will merge HKLM and HKCU. When you are impersonating a user, say in a service, HKCU will still be the settings for the server process and not the impersonated user. This means that users will not be able to override the default behavior when creating COM objects in this manner ².

Couple of links on the issue

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/sysinfo/base/hkey_classes_root_key.asp

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/sysinfo/base/regopenuserclassesroot.asp

limited user account. So what I am really saying is limited user account account. But it just sounds weird to say “with an LUA”.

I know it sounds a bit redundant to say LUA account when LUA stands for ↩
Depending on who the user is, that could be a good thing :) ↩

Singleton Pattern

2004-11-24T00:00:00+00:00

At home I code a lot of singleton classes. Most of this code is boilerplate code that I just write over and over again. Originally I thought that generics would solve this problem. I planned to write a generic singelton of the following form.

public class Singleton<T,K> : K
{
   public static T Instance 
   {
      // ... Singleton code 
   }
}

public class MyFactory : Singleton<MyFactory,MyBaseFactoryClass>
{
}

Unfortunately that doesn’t work because generics in .NET do not support inheritance of that kind (AFAIK). This forced me to find a different route. I decided to code a generic singleton class that operated on static methods. I wanted to get the behavior below.

public class MyFactory
{
   public static MyFactory Instance 
   { 
      get 
      { 
         return Singleton<MyFactory>.GetInstance(); 
      } 
   }
}

This still didn’t solve my problems because this requires that MyFactory have a public constructor. I had to settle for this instead.

public class MyFactory
{
   public static MyFactory Instance 
   { 
      get 
      { 
         return Singleton<MyFactory>.GetInstance(delegate() { return new MyFactory(); } ); 
      } 
   }
}

This makes it very simple to create per process singleton classes. It’s not quite as easy as my original idea but it gets the job done and removes a lot of redundant code. Here’s the code that I’m using for my Singleton class.

public static class Singleton<T>
{
   public delegate T SingletonCreation();

   private static T _instance;

   public static T GetInstance(SingletonCreation del)
   {
      if (_instance == null)
      {
         lock (typeof(Singleton<T>))
         {
            if (_instance == null)
            {
               T temp = del();
               System.Threading.Thread.MemoryBarrier();
               _instance = temp;
            }
         }
      }
      return _instance;
   }
}

For more information on singletons in .NET check out this great entry by Brad Abrams http://blogs.msdn.com/brada/archive/2004/05/12/130935.aspx