The “For in” style of enumeration in Delphi was introduced in Delphi 2005. The idea is to make enumeration over a collection of things easier and cleaner by removing the need for the iterator variable, removing the code for indexing and removing the code for boundary checking. If you think you know everything about enumeration, hold on to your hat. You may be in for a pleasant surprise.
We will start with the basics and then progress to custom enumerators. Seasoned “enumerophiles” may prefer to skip to the “Custom Enumeration over Objects” section. Let’s look at a simple example. Say we’ve got a list of strings, and we want to show them to the user. Here is how we would do it before and after this feature.
Enumeration by index
procedure EnumDemo1; var SomeStrings: array of string; s: string; i: integer; begin Populate( SomeStrings); for i := 0 to Length( SomeStrings) - 1 do begin s := SomeStrings[i]; Dialogs.ShowMessage( s) end end;
Enumeration by for-in
procedure EnumDemo2; var SomeStrings: array of string; s: string; begin Populate( SomeStrings); for s in SomeStrings do Dialogs.ShowMessage( s); end;
Looking at the details of the above, you see that for-in enumeration is not exactly earth-shattering. We have dropped a variable declaration, the for-loop is a little cleaner and indexing is compiler-magiced away. But you’ve got the wrong idea if you think that the key selling point of for-in enumeration is these simple syntactic abbreviations. The real selling point is greater than that.
Stand back from the details of the code, and think about what we are doing at an abstract level. We are looping through a collection of strings and doing something with the members. From this perspective, the mechanism of indexing is a pesky irrelevant detail. What manner of collection it is (array, list, something-else) is a pesky irrelevant detail. Boundary checking? – another irrelevant artefact of implementation. What we have achieved in EnumDemo2() over EnumDemo1(), is the holy grail of language design – a closer alignment of semantics with syntax.
So what can we enumerate over, out-of-the-box? The following VCL types are for-in enumerable? Member types are as you would expect.
Fundamental and compound types – Enumerables
- Dynamic arrays
- Static arrays
- string/ ansistring/ utf8string
Sets are a special case. Enumeration only visits members that are in the set, as opposed to the base range.
- A large suite of RTTI properties.
Interface pointer Enumerables
So what is missing from Delphi out-of-the-box for-in enumeration support?
Perhaps in a future version of Delphi we will get enumeration support for:
- Enumeration types. With some custom declarations, enumeration of “enumerated types” has been solved by Jeroen W. Pluimers
- Set inversion. Currently, there is no way to for-in enumerate over members NOT in set. Wouldn’t it be handy if we had a generic function to invert sets.
- Character Streams. Ever wanted to parse a text file character by character? Wouldn’t it be nice if we could use for-in?
- Reverse order. Have you ever wanted to traverse in reverse order? How useful would this be?
- TDataset – How many times have you written a while-not-eof loop? For us older developers, the number is probably astronomical.
- Predicates. Ever wanted to traverse a collection, but only for those members that met some specified condition. Sure we can do this with a simple if statement, but this pattern is so common, the thought of an XSLT style syntax mechanism for predicates is enticing. Imagine if we could do something like FantasyEnum4()
Predicated enumeration as it is now…
procedure EnumDemo3( Collection: TObjectList<TSomeClass>); var Member: TSomeClass; begin for Member in Collection do if SomeCondition( Member) then Member.SomeAction end;
Fantasy enumeration supporting predicates
procedure FantasyEnum4( Collection: TObjectList<TSomeClass>); var Member: TSomeClass; begin for Member in Collection ! SomeCondition( Member) do Member.SomeAction end;
Generics and Enumeration
Generic classes and for-in enumeration are a good match-up. Take a look at the syntactic synergy in EnumDemo3 above. If we wanted to do the same in Delphi 7, we would probably write something like EnumDemo5() below…
procedure EnumDemo5( Collection: TObjectList); var i: integer; Member: TSomeClass; begin for i := 0 to Collection.Count - 1 do begin Member := Collection[i] as TSomeClass; if SomeCondition( Member) then Member.SomeAction end end;
Custom Enumeration over objects
So much for the basics. Now here is where it gets really interesting. With some customisation you can for-in enumerate over any object of any class.
You can enumerate over objects, records and interface pointers. The enumerators also can either be objects, records or interface pointers. And the collection members of the enumeration can by any type including fundamentals, compounds, objects, interface pointers, anonymous functions, even class references and type references.
When the compiler sees a code like
for Member in Collection do
.. and Collection is an object, it compiles by the following algorithm..
- Look for a public function of name GetEnumerator(). The function return type must be a class, record or interface pointer type. Protected and below methods do not count. Class functions are also discounted.
- If class, Examine the declaration of the class. There must be public non-class function MoveNext(), with signature…
function MoveNext: boolean;
- Also in this class, there must be a public property named Current. The read accessor method for Current may be any thing, even strict private.
- The type for the current property must be equal to the declared type of the enumerated value (‘Member” in our above example), or, if the type is an object type, the current class type may be a descendant of declared member class. If the member type is an interface type, then the current property interface type may be an interface descendant of the declared member interface type. It is not required for interface types to have syntactically bound guids.
- The MoveNext() and Current are utilised as follows…
With declarations …
TMyEnumerator = object strict private function GetCurrent: TMemberClass; public property Current: TMemberClass read GetCurrent; function MoveNext: Boolean; end; TMyCollection = object public function GetEnumerator: TMyEnumerator; end;
…this procedure …
procedure EnumDemo6a( Collection: TMyCollection); var Member: TMemberClass; begin for Member in Collection do SomeAction( Member) end
… is compiled as if we wrote instead…
procedure EnumDemo6b( Collection: TMyCollection); var Member: TMemberClass; Enumerator: TMyEnumerator; begin Enumerator := Collection.GetEnumerator; try while Enumerator.Next do begin Member := Enumerator.Current; SomeAction( Member) end finally Enumerator.Free end end
If the Enumerator is an interface pointer, then instead of being freed, it is released (reference count decremented). If the Enumerator is a record, then instead of being freed, it is record finalized and de-scoped.
If you are making custom enumerators, you may find it convenient to leverage these base classes and interface pointer types from the VCL…
IEnumerator = interface(IInterface) function GetCurrent: TObject; function MoveNext: Boolean; procedure Reset; property Current: TObject read GetCurrent; end; IEnumerable = interface(IInterface) function GetEnumerator: IEnumerator; end; IEnumerator = interface(IEnumerator) function GetCurrent: T; property Current: T read GetCurrent; end; IEnumerable = interface(IEnumerable) function GetEnumerator: IEnumerator; end; TEnumerator = class abstract protected function DoGetCurrent: T; virtual; abstract; function DoMoveNext: Boolean; virtual; abstract; public property Current: T read DoGetCurrent; function MoveNext: Boolean; end; TEnumerable = class abstract protected function DoGetEnumerator: TEnumerator; virtual; abstract; public function GetEnumerator: TEnumerator; end;
Custom Enumeration over records and interface pointers.
You can enumerate over records and interface pointers too. It works the same way as classes, except that the restriction that the GetEnumerator method be public is not applicable.
Custom Enumeration with helpers
There is a natural synergy between class helpers and custom enumeration. See the next section for examples.
Some incredibly useful examples
Parse a character stream, line by line
With a class helper and custom enumeration, we can conveniently pass a text stream, line by line (or character by character just as easily), like so…
procedure ParseLines( TextStream: TStream); var Line: string; begin for Line in TestStream do ParseLine( Line) end;
One possible implementation to achieve this (with a UTF-8 encoded stream) might be….
TTextStreamHelper = class helper for TStream public function GetEnumerator: TEnumerator; end; function TTextStreamHelper.GetEnumerator: TEnumerator; begin result := TLineReader.Create( self); end; TLineReader = class( TEnumerator) private coTextStream: TStream; ciPosition: Int64; csCurrent: string; protected function DoGetCurrent: string; override; function DoMoveNext: Boolean; override; public constructor Create( poTextStream: TStream); end; constructor TLineReader.Create( poTextStream: TStream); begin coTextStream := poTextStream; ciPosition := coTextStream.Position; csCurrent := '' end; function TLineReader.DoGetCurrent: string; begin result := csCurrent end; function TLineReader.DoMoveNext: Boolean; var Ch, Ch2: Ansichar; sWorking: UTF8String; L: integer; begin sWorking := ''; result := False; while coTextStream.Read( Ch, 1) = 1 do begin result := True; if (Ch <> #13) and (Ch <> #10) then begin L := Length( sWorking) + 1; SetLength( sWorking, L); sWorking[L] := Ch; // Adds a code-unit NOT a character! end else begin if (coTextStream.Read( Ch2, 1) = 1) and (((Ch = #13) and (Ch2 <> #10)) or ((Ch = #10) and (Ch2 <> #13))) then coTextStream.Seek( -1, soCurrent); break end; end; csCurrent := UTF8ToString( sWorking) end;
See my previous blog entry Dances with XML
I haven’t yet developed a class helper for traversing a TDataSet, but it strikes me as something that would be incredibly convenient. I invite the reader to have a go. Tell us about your efforts in the comment stream.
Third party support
Mike Lischke’s TVirtualTree, (a tree and list view component), is probably the worlds’s most popular third party component, and for good reason. TVirtualTree supports for-in enumeration.
I suspect that most readers of this blog are also readers of Nick Hodge’s “A man’s got to know his limitations“ blog. In this entry, Nick introduces the Delphi Spring Framework’s extention to IEnumerable. The entry is worth reading, including Nick’s answer to the originating question on StackOverflow.
- One would expect that the member variable would be syntactically required to be a local variable. Surprisingly this is not the case. However, I advise against non-local member variables, as this is contrary to the semantics.
- One would expect that member variables should not be capturable by closures. And this too is not the case. Again, don’t do it. The outcome would be just too messy.
- Within the loop, the member variable cannot be assigned to.
- The member variable can be referenced, both before and after the loop. Unlike a for-to loop variable, it’s value is indeed defined after loop exit, provided that the loop has passed at least one iteration.
Set Inversion: There is a whole thread about that here: http://objectmix.com/delphi/402305-invert-set-members.html
Character Streams: Yup, anything where you have a loop like `while not Expression do begin …. increment; end;` will be more readable with a for-in construct.
Reverse order: which things would you like to reverse?
TBits: Francois Piette wrote a nice article on that this month: http://francois-piette.blogspot.nl/2013/01/original-method-to-iterate-bits-within.html
He also wrote a nice piece on enumerators for containers: http://francois-piette.blogspot.nl/2012/12/writing-iterator-for-container.html
TDataSet: I also wrote a TDataSetEnumerator, just not blogged about it yet.
Predicates and XSLT: I remember a blog about something like that, with incredibly smart code, just forgot who wrote it (;
About the reverse order: Consider the situation where you traverse a list and the visitation action on the list member may potentially result in the member being removed from the list. In this situation, it is usually considered safer and simpler to traverse in reverse order. There are a dozen examples in the VCL where lists are traversed in reverse order precisely because of this reason.
About TDataSetEnumerator. When you do write the blog entry, please insert a link in this comment feed. I would be most interested.
I’m pretty sure that the TDataSetEnumerator code is in the example source code of my “Smarter Code with Databases and Data Aware Controls” for which the download links are: http://wiert.me/2009/04/24/spoken-coderage-iii-december-1-5-2008-on-delphi-database-and-xml-related-topics/
I’m a tad occupied right now (our stove broke down, so need to bring foward of selecting and buying a new one), otherwise I’d have checked myself if the sources are there.
There are couple of issues to using for-in for line parsing though, the main one being that it falls apart as soon as you want to be able to do thing like reporting an error with the line number at which the error occurred…
You end up either having to maintain the line number separately (bad) or exposing the current line number in the iterator (good), which means using the iterator class/interface directly in a while loop (and not for-in).
Also using for-in to process items in a collection is effectively limited to simple cases: since you have no control over the direction of the enumeration, you are unable to perform tasks like deletion, insertions or exchange/move items.
With the out-of-the-box enumerators, your points are bang-on.
With custom enumerators, all of these issues go away with at a surprisingly low cost. The critical question will always be: “When is it worth it, to invest in a custom enumerator?”. It’s a hard one to answer.
And then one could argue that a simple index-based loop or a while loop are more readable than a custom enumerator 🙂
Cool language features are those that improve maintainability and code readability, and don’t promote over-engineering, at some point Enumerator become just an extra layer of complexity rather than being facilitators.
Also there can be something philosophically wrong to exposing as enumerations constructs that aren’t designed for sequential access (like dictionaries or trees).
Some time ago I have written an article showing the implementaion of a dataset enumerator:
Really nice work there Uwe!
That was nice coverage of forin, thx.
Looks like some of your <T> parts were removed fro mthe listing, probably by tge code formatter, thinking they were tags.
Thanks. Correction made.
By the improvement of Delphi, I see that it’s more likely to become an alternative of C# language. It changed from for-to to for-in and in the latest XE4, StringBuilder is recommended to do string handling rather than original functions. You’ll see string.toint() and int.tostring() in the future. TArray will replace Array of T.
Frankly, if it comes closer to C#, why do I have to learn Delphi anymore?
Just some thoughts.