Collection functions

A common use of functional-style programming is applying transformative functions to collections. For example, I might have a List of items, and I want to transform each item a certain way. There are a number of such functions that are commonly found, in one form or another, in languages that allow functional programming. For example, Seq in Scala provides a map() function, while a method of the same name can be found in Java’s Stream class.

Keeping these functions/methods straight can be difficult when starting out, so I thought I’d list out some of the common ones, along with a quick description of their purpose. I’ll use Scala’s implementation primarily, but will try to point out where Java differs. Hopefully the result will be useful for users of other languages as well.

flatten()

Purpose: Takes a list of lists/sequences, and puts each element in each list/sequence into a single list
Result: A single list consisting of all elements
Example:
scala> val letters = List( List(“a”,”e”,”i”,”o”,”u”), List(“b”,”d”,”f”,”h”,”k”,”l”,”t”), List(“c”,”m”,”n”,”r”,”s”,”v”,”w”,”x”,”z”), List(“g”,”j”,”p”,”q”,”y”) )
letters: List[List[String]] = List(List(a, e, i, o, u), List(b, d, f, h, k, l, t), List(c, m, n, r, s, v, w, x, z), List(g, j, p, q, y))
scala> letters.flatten
res19: List[String] = List(a, e, i, o, u, b, d, f, h, k, l, t, c, m, n, r, s, v, w, x, z, g, j, p, q, y)
scala> letters.flatten.sorted
res20: List[String] = List(a, b, c, d, e, f, g, h, i, j, k, l, m, n, o, p, q, r, s, t, u, v, w, x, y, z)

map()

Purpose: Applies a function that transforms each element in a list.
Result: Returns in a list (or stream) consisting of the transformed elements. The resulting elements can be of a different type than the original elements.
Example:
scala> val nums = List(1, 2, 3, 4, 5)
scala> val newNums = nums.map(n => n * 2)
newNums: List[Int] = List(2, 4, 6, 8, 10)
scala> val newStrs = nums.map(n => s”number $n”)
newStrs: List[String] = List(number 1, number 2, number 3, number 4, number 5)

Scala note: Often when working with Futures, you’ll see map() applied to a Future. In this case, the map() method–when provided with a mapping function for the value returned by the Future, returns a new Future that runs once the first Future completes.

flatMap()

Purpose: Takes a sequence/list (A) of smaller sequences/lists (B), applies the provided function to each of those smaller sequences (B), and places the result of each into a single list to be returned. A combination of map and flatten.
Result: Returns a single list (or stream) containing all of the results of applying the provided function to (B).
Example:
scala> val numGroups = List( List(1,2,3), List(11,22,33) )
numGroups: List[List[Int]] = List(List(1, 2, 3), List(11, 22, 33))
scala> numGroups.flatMap( n => n.filter(_ % 2 == 0) )
res8: List[Int] = List(2, 22)

fold()

Purpose: Starts with a single T value (call it v), then takes a List of T and compares each T element to v, creating a new value of v on each iteration. The order of iteration is non-deterministic. Note: This is very similar to the reduce() method in Java streams).
Result: A single T value (which would be the final value of v as described above)
Example:
scala> val nums = List(1,2,3,4,5,6,7,8,9)
nums: List[Int] = List(1, 2, 3, 4, 5, 6, 7, 8, 9)
scala> nums.fold(0) ((a,b) => if (a % 2 == 0) a else b)
res25: Int = 0

foldLeft()

Purpose: Like fold(), always iterating from the left to the right.
Result: A single T value
Example:
scala> val nums = List(1,2,3,4,5,6,7,8,9)
nums: List[Int] = List(1, 2, 3, 4, 5, 6, 7, 8, 9)
scala> nums.foldLeft(-1) ((a,b) => if (a % 2 == 0) a else b)
res27: Int = 2

foldRight()

Purpose: Like fold(), always iterating from the right to the left.
Result: A single T value
Example:
scala> val nums = List(1,2,3,4,5,6,7,8,9)
nums: List[Int] = List(1, 2, 3, 4, 5, 6, 7, 8, 9)
scala> nums.foldRight(-1) ((a,b) => if (b % 2 == 0) b else a)
res28: Int = 8

Business Logic for Scala/Play apps

As I’d mentioned previously, I’m a fairly seasoned Java developer who is making a foray into Scala and its ecosystem (including the Play framework, as well as Akka).

One thing that’s struck me about Play is that there doesn’t seem to be a prescribed pattern for how to handle business logic. This is in contrast to a typical Java EE application (or Spring-based Java web application). With the latter, you’d typically find applications divided into the following:

  • MVC Layer (or Presentation Layer) – This layer would typically contain:
    • UI Models: POJOs designed to transfer values from the controllers to the UI templates
    • Views: files containing markup mixed with UI models, to present data to the user. These would typically by JSPs, or files of some other tempting language such as Thymeleaf, Handlebars, Velocity, etc. They might also be JSON or XML responses in the case of single-page applications
    • Controllers: Classes designed to map requests to business logic classes, and transforming the results into UI Models.
  • Business Layer – This layer would typically contain:
    • Managers and/or Facades: somewhat coarse-grained classes that encapsulate business logic for a given domain; for example, UserFacade, AccountFacade, etc. These classes are typically stateless singletons.
    • Potentially Business Models: POJOs that describe entities from the business’ standpoint.
  • Data Access Layer – This layer would typically contain:
    • DAOs (Data Access Objects): Fine-grained, singleton classes that encapsulate the logic needed to for CRUD operations on database entities
    • Database Entities: POJOs that map to database tables (or Mongo collections, etc, depending on the data source).
By contrast, Play applications seem to typically have only the MVC Layer. Occasionally I’ll see examples with a utils/ folder, but I’ve yet to see any examples with what I would consider an entirely separate business layer or data access layer.
Clearly I could Business and Data Access Layers for my Play application if I wanted to. But part of learning a new framework is not just learning the mechanics, but also the spirit of the framework. So here are a few thoughts I’ve had on the subject:

Rely on Models’ Companion Objects

Scala has the concept of companion objects. A companion object is essentially a singleton instance of a class. For example, I might have a model called User, which looks something like this:
case class User(id: Long, firstName: String, lastName: String email: String)
I would typically create, in the same User.scala file, a companion object like so:
object User {
  def findByEmail(email: String): User = {
    // query the database and return a User
  }
}

As shown above, it’s common for companion objects to contain CRUD operations. So one thought is that we can combine business logic and data access methods in a model’s companion object, treating the companion object as a sort of hybrid manager/DAO.

There are of course a few downsides to this approach. First is that of separation of concerns. If we’re imbuing companion objects with the ability to perform CRUD operations and business logic, then we’ll wind up with large, hard to read companion objects that have multiple responsibilities.

The other downside is that business operations within a companion object would be too fine-grained. Often, business logic spans multiple entities. Trying to choose which entity’s companion object should contain a specific business rule can become cumbersome. For example, say I want to update a phone number. Surely, that functionality belongs in a PhoneNumber object. But… what if my business stipulates that a phone number can only contain an area code that corresponds to its owner’s mailing address? Suddenly, my PhoneNumber object must communicate with a User and MailingAddress object.

Use Middleware for Business Logic

As a Java engineer, I’m using to my business logic being encapsulated within stateless Spring beans. These beans exist in the Spring container (i.e. application context). They are injected into, for example, controllers, who in turn invoke methods on the bean to cause some business operation to concern.

Play ships with Akka our of the box. So I wonder… would a framework like Akka suffice? Presumably I can create an actor hierarchy for each business operation, thus keeping the operations centralized and well-defined. I’m just delving into Akka, so I’m not sure how viable of a solution that would be. My sense is that, at best, I’d be misusing Akka somewhat with this approach. Moreover, I suspect I’m trying to shoehorn a Spring-application paradigm into a Play application.

Let Aggregate Roots Define Business Logic

I’ve coincidentally been reading a lot Martin Fowler‘s blog posts. One idea of his that seems to be picking up traction is that anemic entities–those who are little more than getters and setters–are an anti-pattern. Couple this with the concept of aggregates and aggregate roots presented in Eric Evans’ Domain Driven Design, and I think I might be on the best solution.

The basic premise is that, unlike the layered architecture I described above, with Manager/Facade classes, the domain entities themselves should perform their own business logic. Furthermore, entities should be arranged into aggregates. An aggregate is essentially a group of domain entities that logically form one unit. Furthermore, one of these entities should be the root, to which all other entities are attached. Modifications should only be done through that root.

In my example above about Users, PhoneNumbers and MailingAddresses, those entities would be arranged as an aggregate. User would be the root entity; after all, PhoneNumbers and MailingAddresses don’t simply exist on their own, but rather are associated with a person (or organization). To modify a User’s phone number, I would go through the User rather than directly modifying the PhoneNumber. For example, something like this:

var user: User = Users.findById(123)
user.phoneNumber = “415-555-1212”

rather than this:

var pn: PhoneNumber = PhoneNumbers.findById(789)
ph.value = “415-555-1212”

Using the former approach, my User instance can ensure data integrity; e.g. ensuring that the provided area code is valid.

This, then, may be the best option:

  1. Companion objects handle CRUD data access operations
  2. Entities themselves–organized into aggregates–handle their own business rules
  3. No separate business logic stereotype is called out

Scaling Scala – part 1

It’s time to explore Scala. I still enjoy Java programming, and that’s still what I do for a living. But I have to admit that Scala is intriguing. Plus, it’s good to learn new languages every so often. And as they say, Java is more than a language; it’s also a platform and an ecosystem, one that Scala fits very well into.

I’ve gone through different books and tutorials, but the best way to learn a language is to come up with a task and figure out how to do it. I’ve decided that while I’m learning Scala, I’ll also learn the Play! framework. My task will be a microservice whose purpose is to authenticate users. Although my current employer doesn’t use Scala (at least not directly, although we do use Kafka, Akka, and other technologies written in Scala) my plan is to write a service that could be used by the company. That way I won’t be tempted to cut corners.

As I develop this project, I’ll post the solutions to any issues I encounter along the way. I figure there are probably enough Scala noobs out there who might encounter the same problems. I also figure that some of these issues might be very elementary to developers who are more experienced with Play! and Scala. In that vein, any corrections or suggestions are most welcome!

Maven Repositories (and the Oracle driver)

I started a few days ago, and hit two issues rather quickly. The first stems from my company’s use of Oracle as its RDBMS; that’s where we store user credentials. So of course I would need to read from Oracle in order to authenticate users.

As I understand it, Play! makes use of SBT (the Simple Build Tool), which is developed by Typesafe, and is used to manage Play! projects (and other Scala-based frameworks). SBT is analogous to Maven for pure Java projects. In fact, SBT makes use of Ivy for dependency management; Ivy, in turn, makes use of Maven repositories.

So I figured I’d need to pull the Oracle JDBC driver from Maven Central. Play! projects are created with a build.sbt file in the project’s root directory, and that’s where dependencies are listed. We use ojdbc6 (the Oracle JDBC driver for Java 6), so our POM entry looks like this:

<dependency>
    <groupId>com.oracle</groupId>
    <artifactId>ojdbc6</artifactId>
    <version>11.2.0.3</version>
</dependency>

To add that to build.sbt, it would look like this:

libraryDependencies += "com.oracle" %% "ojdbc6" % "11.2.0.3"

I added that to build.sbt, and was confronted with errors stating that that dependency couldn’t be resolved. Turns out that due to licensing reasons, Oracle does not allow its JDBC driver in any public repos. So the solution is to simply download the JAR file from Oracle, create a lib/ directory in the Play! project’s root, and add the JAR there.

Admittedly, that was as much an issue of the ojdbc6 driver than with Play! itself, but I thought it worth documenting here.

Artifactory (or Nexus, if you prefer)

Next up was the issue of artifacts that my company produces. Much of our code is encapsulated in common JAR files and, of course, hosted in our own internal (Artifactory) repository. Specifically, the domain class that I would be pulling from the database contains, among other fields, and enum called Type (yes, I know… that name was not my doing!) which was located in a commons module. I could’ve created a Scala Enumeration for my project, or just skipped the field, but I wanted to demonstrate the interoperability between new Scala projects and our existing Java code.

So I’d have to point SBT to the correct location to find the commons module. I found bits and pieces online on how to do it; here’s the solution that I ultimately pieced together:

(1) SBT had already create a ~/.sbt/0.13/ directory for me. In there, I created a plugins/ subdirectory and with there a credentials.sbt file with these contents:

credentials += Credentials(Path.userHome / “.ivy2” / “.credentials”)

(2) Within the existing ~/.ivy2/ directory, created a .credentials file with these contents:

realm=[Artifactory Realm]
host=company.domain.com
user=[username]
password=[password]

(3) Add the repository location in ~/.sbt.repositories like so:

…
my-maven-proxy-releases: https://artifactory.tlcinternal.com/artifactory/LC_CENTRAL
…

(4) Added the following line in ~/.sbtconfig to tell SBT where to find the credentials:

export SBT_CREDENTIALS=”$HOME/.ivy2/.credentials”

I’m not sure why we need both step 1 and 4, but both seem to be required.

Once I restarted my Play! application (this was one case where hot-deployment didn’t seem to work) I was able to use the commons module in my Play! app.

Spring Data, Mongo, and Lazy Mappers

In a previous post, I mentioned two things that every developer should do when using Spring Data to access a MongoDB datastore. Specifically, you should be sure to annotate all of your persistent entities with @Document(collection=”<custom-collection-name>”) and @TypeAlias(“<custom-type>”). This decouples your Mongo document data from your specific Java class names (which Spring Data will otherwise closely couple by default) making things like refactoring possible.

With my particular application, however, I ran into an additional problem. Let me recap. My application is a drawing application of sorts. Drawings are are modeled by, well, a Drawing class. A Drawing can contain multiple instances of Page, and within each Page, multiple Shape objects. Shape, in turn, is an abstract class, containing a number of subclasses (Circle, Star, Rectangle, etc).

For our purposes, let’s focus on the relationship between a Page and its Shapes. Here’s a snippet from the Page class:

@Document(collection=”page”)
@TypeAlias(“myapp.page”)
public class Page extends BaseDocument {

    @Id 
    private String id;

    @Indexed
    private String drawingId;

    private List<Shape> shapes = new ArrayList<Shape>();

    // ….


}

First not that I’ve annotated this class so that I have control over the name of the collection that stores Page documents (in this case, “page”), and so that Spring Data will store an alias to the Page class (in this case, “my app.page”) along with the persisted Page documents, rather than storing the fully-qualified class name.

Also of importance here is that the Page class knows nothing about any specific Shape subclasses. This is important from an OO perspective, of course; I should be able to add any number of Shapes to my app’s ecosystem, and the Page class should continue to work with no modifications.

Now let’s look at my Shape class:

public abstract class Shape extends BaseDocument {

 

    @Id
    private String id;

 

    @Indexed
    private String pageId;

 

    // attributes
    private int x;

    private int y;

    // …

}

Nothing surprising here. Note that Shape has none of the SpringData annotation; that’s because no concrete instance of Shape will be persisted along with any Pages. It is abstract, after all. Instead, a Page will contain instances of Shape subclasses. Let’s take a look at one such subclass:

@Document(collection=”shape”)
@TypeAlias(“myapp.shape.star”)

public class Star extends Shape {

    private int numPoints;
    private float innerRadius;

    private float outerRadius;

}

The @Document(collection=”shape”) annotation is currently unused, because per my app design, any Shape subclass instance will always be stored as a nested collection within a Page. But it would certainly be possible to store different shapes directly into a specific collection.

The @TypeAlias annotation, however, is very important. The purpose of that one is to tell Spring Data how to map the different Shapes that it finds within a Page back into the appropriate class. After all, if a Page containing a nine-point star is persisted, then when it’s read back in, that star had better be mapped back into a Star class, not a simple Shape class. After all, Shape itself knows nothing about number of points!

Feeling pretty happy with myself, I tried out my code. Upon trying to read my drawings back in, I began getting errors of this type:

org.springframework.data.mapping.model.MappingInstantiationException: Could not instantiate bean class [com.myapp.documents.Shape]: Is it an abstract class?; nested exception is java.lang.InstantiationException

Indeed, Shape is an abstract class, and so indeed, it cannot be directly instantiated. But why was Spring Data trying to directly instantiate a Shape? I played around, tweaked a few things, but nothing fundamentally changed. I checked StackOverflow and the Spring forums. Nothing. So it was time to dig into the documentation.

As with most typical Spring Data/Mongo apps, mine was configured to use a bean of type org.springframework.data.mongodb.core.convert.DefaultMongoTypeMapper to map persistence documents to and from Java classes:

     <bean id=”mongoTypeMapper” class=”org.springframework.data.mongodb.core.convert.DefaultMongoTypeMapper“>
        <constructor-arg name=“typeKey” value=“_alias”></constructor-arg>

    </bean>

    <bean id=”mappingMongoConverter”
class=”org.springframework.data.mongodb.core.convert.MappingMongoConverter”>
        <constructor-arg ref=“mongoDbFactory” />
        <constructor-arg ref=“mappingContext” />
        <property name=“typeMapper” ref=“mongoTypeMapper“/>
    </bean>

    <bean id=”mongoTemplate” class=”org.springframework.data.mongodb.core.MongoTemplate”>
        <constructor-arg ref=“mongoDbFactory” />
        <constructor-arg ref=“mappingMongoConverter” />

    </bean>

The docs indicated that DefaultMongoTypeMapper was responsible for reading and writing the type information stored with persistent data. By default, this would be a _class property pointing to com.myapp.documents.Star; with my customizations it became an _alias property pointing to may app.shape.star. But if DefaultMongoTypeMapper wouldn’t do the trick, perhaps I needed another mapper.

Looking through the documentation, I found org.springframework.data.convert.MappingContextTypeInformationMapper. Here’s what its Javadocs indicated:

TypeInformationMapper implementation that can be either set up using a MappingContext or manually set up Map of String aliases to types. If a MappingContext is used the Map will be build inspecting the PersistentEntity instances for type alias information.

That seemed promising. If I could replace my DefaultMongoTypeMapper with a MappingContextTypeInformationMapper that could scan my persistent entities and build a type-to-alias mapping, then that should solve my problem. The docs also said something about manually creating a Map, but a) It wasn’t readily apparent how to create a Map myself, and b) I didn’t like that approach; I didn’t want to have to manually configure an entry for any new Shape that might be created.

One problem. You’ll notice above that my DefaultMongoTypeMapper is wired into my MappingMongoConverter by way of the latter’s typeMapper property. In fact, typeMapper is itself of type MongoTypeMapper. While DefaultMongoTypeMapper implements MongoTypeMapper,  MappingMongoConverter does not. Fortunately, DefaultMongoTypeMapper allows you to chain together fallback mappers by way of an internal property, mappers, which itself is a List<? extends TypeInformationMapper>. And as luck would have it, MappingMongoConverter implements TypeInformationMapper.

So I would keep my DefaultMongoTypeMapper, and add a MappingMongoConverter to its mappers list. I modified my spring XML config like so:

  <bean id=”mongoTypeMapper” class=”org.springframework.data.mongodb.core.convert.DefaultMongoTypeMapper”>
<constructor-arg name=”typeKey” value=”_alias”></constructor-arg>
    <constructor-arg name=”mappers“>
        <list>
            <ref bean=“mappingContextTypeMapper“ />
        </list>
    </constructor-arg> 
</bean>

  <bean id=”mappingContextTypeMapper” class=”org.springframework.data.convert.MappingContextTypeInformationMapper“>
      <constructor-arg ref=”mappingContext” />

  </bean>

I redeployed and ran my app.

And I ran into the same exact error. Damn.

At this point, I became concerned that maybe all of the TypeAlias information was completely ignored by SpringData with nested documents, such as my Shapes nested within Pages. So I decided to roll up my sleeves, fire up my debugger, and start getting intimate with the Spring Data source code.

After a bit of debugging, I learned that Spring Data was indeed attempting to determine if any TypeAlias information applied to the Shapes that were being read in for any Page. But in a lazy, half-hearted way.

When I say lazy, I mean that there was absolutely no scanning of entities to search for @TypeAlias annotation like I’d assumed there would be. Everything was done at runtime, as new data types were discovered. The MappingMongoConverter would read my base entity; i.e. a Page document. It would then discover that this document had a collection of things called shapes. Then it would examine the Page class to find the shapes property, and discover that shapes was of type List<Shape>. And finally it would examine the Shape class to determine if it had any TypeAlias data that it could cache for later.

In other words, it was completely backwards from what I needed. This mapper wouldn’t work for me, either.

By this time, I’d developed enough understanding as to what was going on, that creating my own mapper didn’t seem too tough. And that’s what I did. Really, all I needed was a mapper that I could configure to scan one or more packages to discover persistent entities with TypeAlias information, and cache that information for later use.

My class was called EntityScanningTypeInformationMapper, and its full source code is a the end of this post. But the relevant parts are:

  • Its constructor takes a List<String> of packages to scan.
  • It has an init() method that scans the provided packages
  • Scanning a package entails using reflection to read in the information for each class in the package, determining if it is annotated with @TypeAlias, and if so, mapping the alias to the class.

My Spring XML config was modified thusly:

  <bean id=”mongoTypeMapper” class=”org.springframework.data.mongodb.core.convert.DefaultMongoTypeMapper”>
<constructor-arg name=”typeKey” value=”_alias”></constructor-arg>
    <constructor-arg name=”mappers”>
        <list>
            <ref bean=”entityScanningTypeMapper” />
        </list>
    </constructor-arg> 
</bean>

  <bean id=”entityScanningTypeMapper” class=”com.myapp.utils.EntityScanningTypeInformationMapper” init-method=”init”>
    <constructor-arg name=”scanPackages”>
        <list>
            <value>com.myapp.documents.shapes</value>
        </list>
    </constructor-arg> 

  </bean>

I redeployed and retested, and it ran like a champ.

So my lesson is that Spring Data, out of the box, doesn’t seem to work well with polymorphism, which is a shame given the schema-less nature of NoSQL data stores like MongoDB. But it doesn’t take too much effort to write your own mapper to compensate.

Oh, and here’s the EntityScanningTypeInformationMapper source:

public class EntityScanningTypeInformationMapper implements TypeInformationMapper {
    private Logger log = Logger.getLogger(this.getClass());
    private final List<String> scanPackages;
    private Map<String, Class<? extends Object>> aliasToClass;
    public EntityScanningTypeInformationMapper(List<String> scanPackages) {
        this.scanPackages = scanPackages;
    }
    public void init() {
       this.scan(scanPackages);
    }
    private void scan(List<String> scanPackages) {
        aliasToClass = new HashMap<>();
        for (String pkg : scanPackages) {
            try {
                findMyTypes(pkg);
            } catch (ClassNotFoundException | IOException e) {
                log.error(“Error scanning package “ + pkg, e);
            }
        }
    }
    private void findMyTypes(String basePackage) throws ClassNotFoundException, IOException {
        ResourcePatternResolver resourcePatternResolver = new PathMatchingResourcePatternResolver();
        MetadataReaderFactory metadataReaderFactory = new CachingMetadataReaderFactory(resourcePatternResolver);
        String packageSearchPath = ResourcePatternResolver.CLASSPATH_ALL_URL_PREFIX +
                                   resolveBasePackage(basePackage) + “/” + “**/*.class”;
        Resource[] resources = resourcePatternResolver.getResources(packageSearchPath);
        for (Resource resource : resources) {
            if (resource.isReadable()) {
                MetadataReader metadataReader = metadataReaderFactory.getMetadataReader(resource);
                Class<? extends Object> c = Class.forName(metadataReader.getClassMetadata().getClassName());
                log.debug(“Scanning package “ + basePackage + ” and found class “ + c);
                if (c.isAnnotationPresent(TypeAlias.class)) {
                    TypeAlias typeAliasAnnot = c.getAnnotation(TypeAlias.class);
                    String alias = typeAliasAnnot.value();
                    log.debug(“And it has a TypeAlias “ + alias);
                    aliasToClass.put(alias, c);
                }
            }
        }
    }
    private String resolveBasePackage(String basePackage) {
        return ClassUtils.convertClassNameToResourcePath(SystemPropertyUtils.resolvePlaceholders(basePackage));
    }
    @Override
    public TypeInformation<?> resolveTypeFrom(Object alias) {
        if (aliasToClass == null) {
            scan(scanPackages);
        }
        if (alias instanceof String) {
            Class<? extends Object> clazz = aliasToClass.get( (String)alias );
            if (clazz != null) {
                return ClassTypeInformation.from(clazz);
            }
        }
        return null;
    }
    @Override
    public Object createAliasFor(TypeInformation<?> type) {
        log.debug(“EntityScanningTypeInformationMapper asked to create alias for type: “ + type);
        return null;
    }

 

}

Before You Use SpringData and MongoDB

The Upfront Summary

For those who don’t have time to read a long blog post, here’s the gist of this article: always always always annotate your persisted SpringData entity classes with @Document(collection=”<custom-collection-name>”) and @TypeAlias(“<custom-type>”) . This should be an unbreakable rule. Otherwise you’ll be opening yourself up to a world of hurt later.

SpringData is Easy to Get Started With

Like many Java developers, I rely on the Spring Framework. Everything, from my data access layer to my MVC controllers are managed within a Spring application context. So when I decided to add MongoDB to the mix, it was without a second thought that I decided to use SpringData to interact with Mongo.

That was months ago, and I’ve run into a few problems. As it turns out, these particular problems were easy to solve, but it took awhile to recognize what was going on and come up with a solution. Surprisingly little information existed on StackOverflow or the Spring forums for what I’m imagining is a common problem.

Let me explain.

My Data Model

My application is basically an editor. Think of a drawing program, where users can edit a multi-page “drawing” document. Within a drawing’s page, users can create and manipulate different shapes. As a document store, MongoDB is well-suited for persisting this sort of data. Roughly speaking, my data model was something like this (excuse the lack of UML):

  • Drawings are the top-level container
  • A Drawing has one or more Pages
  • A Page consists of many Shapes.
  • Shape is an abstract class. It has some properties shared by all Shape subclasses, such as size, border with and color, background color, etc
  • Concrete subclasses of Shape can contain additional properties. For example, Star has number of points, inner radius, outer radius, etc

Drawing are stored separately then pages; i.e. they are not nested. Shapes, however, are nested within Pages. For example, here’s a snippet from the Drawing class:

public class Drawing extends BaseDocument {

    @Id 
    private String id;

    // ….


}

and the Page class:

public class Page extends BaseDocument {

    @Id 
    private String id;

    @Indexed
    private String drawingId;

    private List<Shape> shapes = new ArrayList<Shape>();

    // ….


}

So in other words, when a user goes to edit a given drawing, we simply retrieve all of the Pages whose drawingId matches the ID of the drawing being edited.

Don’t Accept SpringData’s Defaults!

SpringData offers you the ability to customize how your entities are persisted in your datastore. But if you don’t explicitly customize, SpringData will make do as best as it can. While that might seem helpful up front, I’ve found the opposite. SpringData’s default behavior will invariably paint you into a corner that’s difficult to get out of. I’d argue that, rather than guessing, SpringData should throw a runtime exception at startup. Short of that, every tutorial about SpringData/MongoDB should strongly encourage developers of production applications to tell Spring how to persist their data.

The first default is how SpringData maps classes to collections. Collections are how many NoSQL data stores, MongoDB included, store groups of related data. Although it’s not always appropriate to compare NoSQL databases to traditional RDBMs, you can roughly think of a collection the same way you think of a table in a SQL database.

Chapter 7 of the SpringData/Mongo docs explains how, by default classes are mapped to collections:

The short Java class name is mapped to the collection name in the following manner. The class ‘com.bigbank.SavingsAccount‘ maps to ‘savingsAccount‘ collection name.

So based on my data model, I knew I’d find a drawing collection and a page collection in my MongoDB instance.

Now, I’ve used ORMs like Hibernate extensively. Probably for that reason, I wasn’t content to let my Mongo collections be named for me. So I looked for a way to specify my collection names.

The answer was simple enough. Although not a strict requirement, persisted entities should be annotated with the @org.springframework.data.mongodb.core.mapping.Document annotation. Furthermore, that annotation takes a collection argument in which you can pass your desired collection name.

So my Drawing class became annotated with @Document(collection=”drawing”), and my Page class became annotated with @Document(collection=”page”). The end result would be the same–a drawing and a page collection in Mongo–but I now had control. I specified the collection name simply because it made me feel more comfortable, but it turns out there’s an important, tangible reason to do so (more on that later).

With everything in place, I started testing my app. I created a few drawings, added some pages and shapes, and saved them all to MongoDB. Then I used the mongo command-line tool to examine my data. One thing immediately stuck out. Every document stored in there had a _class property which pointed to the fully-qualified name of the mapped class. For example, each Page document contained the property “_class” : “com.myapp.documents.Page”.

The purpose of that value, as you might guess, is to instruct Spring Data how to map the document back to a class when reading data out. This convention struck me as a little concerning. After all, my application might be pure Java at this point, but my data store should be treated as language-agnostic. Why would I want Java-specific metadata associated with my data?

After thinking about it, I shook off my concern. Sure, the _class property would be there on every record, but if I started using another platform to access the data, then the property could just be ignored. Practically speaking, what could actually go wrong?

What Could Go Wrong

Then one day I decided to refactor my entire application. I’d been organizing my code based on architectural layer, and I decided instead to try organizing it by feature instead. Eclipse of course allowed me to do this in a matter of minutes. So I WARred up my changes, deployed them to Tomcat, and viola! I could no longer read in any of my drawing/page/shape data.

It quickly became clear what the problem was. My data contained _class information that pointed to a now-non-existence fully-qualified class name. Shape was no longer in the com.myapp.documents package.

With the problem identified, what was the solution?

Making it Right

As mentioned above, SpringData offers the @TypeAlias annotation. Annotating a document as such and providing a value tells Spring to store that value, rather than the fully-qualified classname, as the document’s _class parameter.

So here’s what I did:

@Document(collection=”page”)
@TypeAlias(“myapp.page”)
public class Page extends BaseDocument {
    // …
}

Of course, I still couldn’t read any of my existing data in, but moving forward, any new data would be refactor-proof. Fortunately my app was nowhere near production-ready at this point, so deleting all of my old drawings and starting with new ones was no problem. If that wasn’t an option, then I figure I’d have two options:

  1. Change the @TypeAlias value to match the old, fully-qualified class name, rather than the generic myapp.page value. Of course I’d be stuck with a confusing, language-specific value at that point.
  2. Go through each of the affected collections in my MongoDB store and update their _class values to the new, generic aliases. Certainly possible, although a bit scary for my taste as a MongoDB newbie.

One additional improvement could be made at this point. The property in the MongoDB documents is still called _class, but now that’s a bit of a misnomer. I’d prefer something like, well, _alias. This is easy to change. Somewhere in your XML or Java config, you’ve probably specified a DefaultMongoTypeMapper. Simply pass a new typeKey value in the constructor. For example, here’s a snippet from my XML config:

  <bean id=”mongoTypeMapper” class=”org.springframework.data.mongodb.core.convert.DefaultMongoTypeMapper”>
<constructor-arg name=”typeKey” value=”_alias”></constructor-arg>

</bean>

Are We All Set?

It turns out that I immediately ran into another problem. This one is a bit more specific to my particular application, so I’ll describe it in my next article.

Eclipse Run Configurations and VM Arguments

One of the great things about an IDE like Eclipse is how easy it is to run your application on the fly, as you’re developing it. Eclipse uses the concept of a Run Configuration for this. Run configurations define your code’s entry point (i.e. the project and Main class), as well as numerous other aspects of the run, including the JRE to use, the classpath, and various arguments and variables for your application’s use.

If you’re a developer who uses Eclipse, chances are you’ve created your own Run Configuration. It’s simple. Just go to the Run > Run Configurations… menu item. In the dialog that appears, click the New icon at the top left, and create your new configuration.

You can then use the six or so tabs to the right to configure your runtime environment. One of the options you can configure, within the Arguments tab, is the VM arguments to be be passed to your application. For example, one of my web applications can be run in different environments; certain settings will change depending on whether I’m running in development, staging, or production mode. So I pass a -Dcom.taubler.env=x argument when I launch my application (where x might equals dev, stage, or prod). When I run my application in Tomcat, I simply add the argument to my startup script. Similarly, when I run my application through Eclipse, I can the argument to my Run Configuration, within the Arguments tab.

This works great when you have a single Run Configuration, or at least a small number of stable Run Configurations. But I’ve found an issue when running unit tests through Eclipse. It seems that whenever you run a JUnit test in an ad-hoc manner (for example, right-clicking on a test class and selecting Run As > JUnit Test) Eclipse will implicitly create a Run Configuration for that test. This can result in numerous run configs that you didn’t even know you’d created. That in and of itself isn’t a problem. However, if your application code is expecting a certain VM argument to be passed in, how can you pass that VM argument in to your test run?

Initially when I encountered this problem, I found a solution that didn’t scale very well. Essentially I would run a unit test for the first time, allow Eclipse to create the associated Run Configuration, and let the test error out. Then I would open the Run Configurations window, find the newly-created configuration, click into the Arguments tab and add my -D argument. At that point, I could re-run my test successfully.

It turns out there’s a better way. You can configure Eclipse to always, by default, include one or more VM arguments whenever it launches a particular Java runtime environment. To do this, open Eclipse’s Preferences window. Expand the Java section and select Installed JREs. Then in the main content window, select the JRE that you are using to run your project, and click the Edit… button. A dialog will appear, with an entry field labeled Default VM arguments. That’s where you can enter your VM argument;for example, -Dcom.mycompany.myarg=abc123. Close the window, and from then on, any unit tests you run will automatically pass that argument to the VM.

There are of course a few downsides. The first is that, as a preference, this setting won’t be included with your project (of course, this could also be seen as a benefit). Secondly, this preference is tied to a specific JRE, so if you test with multiple JREs, you’ll need to copy the argument for all JREs. Still, it’s clearly a workable, scalable solution.

Easily Manage Properties in your Java / Spring application

Question to the Java developers out there: properties are easy, right? All you need to do is to type up some key/value pairs in a simple file, read that file in when your application starts up (Java makes that a no-brainer), and refer to the appropriate property in your application code. If you’re using Spring, it’s even easier; just configure your beans in your app context XML file.

Well, that’s fine for a simple application that you’re deploying to your local test server. But for a production ready app, things are a little more complication. There are typically two things you’ll need to be able to deal with:

  1. You’ll like have different servers–test, staging, production, etc–each with their own different properties. For example, your test server will likely have different database credentials and connection URLs than your production server.
  2. You won’t want to store all of your properties in plaintext.
What we need is a way to separate the properties that differ across environments. Furthermore, we want to be able to encrypt some of these properties in case our application falls into the wrong hands.

Separate properties by environment

Let’s look at how we’d typically set properties in a Spring web application. For our purposes here, we’ll focus on setting our datasource’s properties: username, password, url, and the JDBC driver class name.
These would typically be set in your spring config file like so:
<bean id=“dataSource”
class=“org.springframework.jdbc.datasource.DriverManagerDataSource”>
<property name=“driverClassName” value=“com.mysql.jdbc.Driver” />
<property name=“url” value=“jdbc:mysql://127.0.0.1:3366/my_db” />
<property name=“username” value=“joe” />
<property name=“password” value=“foobarbaz” />
</bean>
Of course, any of these properties might change as you deploy to a different environment. Your production database might be running on its own server, for example. And you certainly wouldn’t want to use the same credentials in production as you do on your test server… right?
So our first step is to externalize these properties into a separate file. Let’s create a file called db.properties. This file should go somewhere in our classpath; for example, in com/mycompany/config in your compile target directory path. This file will contain our data source values, like so:
db.driver=com.mysql.jdbc.Driver
db.url=jdbc:mysql://127.0.0.1:3366/my_db
db.username=joe
db.password= foobarbaz
Now, how can we use this properties file? For this, Spring provides a handy class, org.springframework.beans.factory.config.PropertyPlaceholderConfigurer. This class takes a list of property file locations. It will read all of the files in the list, and store their properties to use as placeholder replacements.
To use a PropertyPlaceholderConfigurer, simply declare it as a bean:
<bean id=“propertyPlaceholderConfigurer”
class=“org.springframework.beans.factory.config.PropertyPlaceholderConfigurer“>
<property name=“locations”>
<list>
<value>classpath:com/mycompany/config/db.properties
</value>
</list>
</property>
</bean>
Now we can replace are hard-coded properties in our spring config file with these placeholders:
<bean id=“dataSource”
class=“org.springframework.jdbc.datasource.DriverManagerDataSource”>
<property name=“driverClassName” value=“${db.driver}” />
<property name=“url” value=“${db.url}” />
<property name=“username” value=“${db.username}” />
<property name=“password” value=“${db.password}” />
</bean>
At runtime, Spring will replace the ${…} tokens with their property values.
That works great, but what we really want is separate config files, one per environment. To do this, we copy our db.properties file to db-dev.properties and db-prod.properties (this assumes that we have two environments, development and production; of course, make as many copies as you have distinct environments.) Those files should of course continue to reside on our classpath. And of course, the values of each of the four properties should be changed to match the data source settings for the specific environment.
At runtime, we’ll want Spring to read the current environment’s db properties file. Fortunately, when it performs placeholder replacements, Spring will look for values not just read in by the PropertyPlaceholderConfigurer, but also system properties and VM arguments as well. So what we can do is to pass a VM argument on our startup script. This value will define the current environment. So in our dev environment we’d add this to our startup script:
    -Dcom.mycompany.env=dev
while in production we’d add this:
    -Dcom.mycompany.env=prod
Last, we adjust our PropertyPlaceholderConfigurer location to tell it which configuration file to read in.
<bean id=“propertyPlaceholderConfigurer”
class=“org.springframework.beans.factory.config.PropertyPlaceholderConfigurer“>
<property name=“locations”>
<list>
<value>classpath:com/mycompany/config/db-${com.mycompany.env}.properties
</value>
</list>
</property>
</bean>
The highlighted token above will be replaced with the value of com.mycompany.env set in our startup script. As a result, our PropertyPlaceholderConfigurer will read in the environment-appropriate file, and our dataSource bean will be populated with the environment-appropriate values.

Encrypt your credentials

Our second goal is to keep from bundling our database credentials in plaintext when we package and distribute our web application. To do this, we are going to override Spring’s PropertyPlaceholderConfigurer with our own implementation. This implementation will do one additional thing: when it encounters a property whose name ends with “-enc”, it will assume that the value is encrypted, and therefore decrypt the value just after reading it in.
First, let’s assume that we have written a class called DataEncrypter, containing two methods:
public String encrypt(String rawValue);

public String decrypt(String encryptedValue);

Their functions should be obvious; encrypt() takes a plaintext String and converts it to an encrypted String; decrypt() reverses the process. Both methods should of course rely on the same key(s). This tutorial will skip what an actual implementation of each method might look like; that’s for the security experts to provide. For information, look up information about the javax.crypto package. Instead, we’ll assume that a good implementation has been written.
The first thing we’ll want to do to encrypt our database credentials with our DataEncrypter instance, and put those encrypted values into our properties files. The most straightforward way to do this is to simply create a Java class with a main(String[] args) method, which uses our DataEncrypter to encrypt a value passed as args[0]:
public static void main(String[] args) {
DataEncrypter encrypter = …; // obtain an instance
System.out.println( encrypter.encrypt(args[0]) );
}
Run that class once per property that you want to encrypt; alternatively, you can modify it to expect multiple properties in args, and encrypt them all in one run. Next, we’ll swap those values in to the properties file in place of their plaintext version. We’ll also rename to property with an “-enc” suffix.
db.driver=com.mysql.jdbc.Driver
db.url=jdbc:mysql://127.0.0.1:3366/my_db
db.username-enc=f7BjAyDkWSs=
db.password-enc=4Q7xTCr5hZC9Ms6iTSjG3Q==
So how will those values be decrypted? We’ll need to create our own subclass of PropertyPlaceholderConfigurer (which we’ll call DecryptingPropertyPlaceholderConfigurer). PropertyPlaceholderConfigurer contains one important method that we’ll want to override:
protected void convertProperties(Properties props)
This method read in the properties from any file found in its list of file locations. We’ll want to look for any properties whose name ends with “-enc”, and invoke our DataEncrypter’s decrypt() method:
public class DecryptingPropertyPlaceholderConfigurer extends PropertyPlaceholderConfigurer {
private static final String SUFFIX_ENC = “-enc”;
private final DataEncrypter encrypter = …; // obtain an instance
private final Logger log = LoggerFactory.getLogger (getClass ());
@Override
protected void convertProperties(Properties props) {
// props refers to the contents of a single properties file
Enumeration<?> propertyNames = props.propertyNames();
while (propertyNames.hasMoreElements()) {
String propertyName = (String)propertyNames.nextElement();
String propertyValue = props.getProperty(propertyName);
// look for names such as password-enc
if (propertyName.endsWith(SUFFIX_ENC)) {
try {
String decValue = encrypter.decrypt(propertyValue);
propertyValue = decValue;
// normalize the property name by 
// removing the “enc-” prefix
propertyName = propertyName.substring(
0, propertyName.length() – SUFFIX_ENC());
} catch (EncryptionException e) {
log.error( String.format(
  “Unable to decode %s = %s”, propertyName, propertyValue), e);
throw new RuntimeException();
}
}
props.setProperty(propertyName, propertyValue);
}
}
}
Note that we strip the “-enc” suffix from each encoded property that is encountered. That we, we can continue to refer to our data source password, for example, as db.password rather than db.enc-password. Note also that if we encounter an error decrypting a property, we log the issue and throw a RuntimeException. This is generally the correct thing to do; we don’t want our application to run with partially-incorrect properties. One thing to note here is that you might want to remove the logging of the propertyValue. A common encryption problem is one where someone forgot to encrypt the value before adding it to the properties file. In such a case, you probably would not want the plaintext value hanging out in your log file.
The last thing to do is simply plug our subclass into our Spring config file:
<bean id=“propertyPlaceholderConfigurer”
class=“com.mycompany.DecryptingPropertyPlaceholderConfigurer”>
<property name=“locations”>
<list>
<value>classpath:com/mycompany/config/db-${com.mycompany.env}.properties
</value>
</list>
</property>
</bean>
Now, an important caveat is that this is not an end-all, be-all solution to securing your credentials. This approach is only as strong as your encryption algorithm, and it will never ensure 100% security. It’s not a substitute for simply ensuring that your source code and your webapp bundle remain only in your hands (or for taking other measures such as restricting access to your database server, not reusing passwords, etc). But should someone else get get ahold of your credentials, this approach should at least buy you enough time to change your credentials to prevent unauthorized access.

Done!

Now that you’ve set up your infrastructure, it should be easy to see how to set additional properties, whether in the same files we’d created above or in new files that you might create. Managing properties is once again very easy, but now it’s also very flexible and a tad more secure than it was before.

Profiles and certs and devices… oh hell!

Has it really been more than two years since I’ve written a post? It’s time to start writing again!

I thought that I would kick things off with something that’s cropped up again and again over those past 2+ years. And that is dealing with provisioning profiles when testing and deploying iOS apps. There may seasoned, full-time iOS engineers out there that have all of this down-pat. But for me, someone who doesn’t focus all of his attention on iOS anymore, this process can be perpetually confusing. I’ve tended to learn just enough to get things working, and when things would later mysteriously break, I’d flail around for awhile until things started working again.

If you’re like me, then read on. It’s time to figure this stuff out once and for all.

Provisioning Profiles

The first thing you learn is that everything centers around Provisioning Profiles. Provisioning profiles are really an aggregation of other elements that together determine whether an app can be built and run on a device. There are two types of provisioning profiles: Development and Distribution. As their names suggest, Development profiles are used to test ad hoc builds as you’re developing you app; Distribution profiles are used to distribute apps to the App Store.

We’ll focus on Development profiles here. Development profiles aggregate three things:

  1. Your App ID; for example, com.mycompany.mymobileapp. Each profile can contain only one app ID.
  2. A list of devices; specifically, device UUIDs. To install a build onto a device for ad hoc testing, that device’s UUID must be present in the development profile.
  3. A list of developers certificates that can be used to sign the profile.

App IDs

As mentioned above, each profile can contain only one app ID, so if you have multiple app IDs, you’ll need a separate profile for each. For example, my current company uses two app IDs: one for use when hitting our staging/test server, and one for our production server. So we need two separate development profiles.
Well, that’s not entirely true. You can get away with using wildcards if your app does not use certain features such as Apple Push Notifications. For example, if you have a simple app that uses the App ID com.mycompany.mymobileapp.testing when hitting a testing server, and com.mycompany.prod when hitting a production server, you can create a single Development profile with the App ID of
com.mycompany.*

Devices

To help enforce the “walled garden” App Store distribution model, development builds can only be installed on a limited number of devices. Apple allows each developer account up to 100 devices–identified by their UUIDs–to be registered. These UUIDs are stored in your development profile, ensuring that the profile can only be used on one of those devices.

Certificates

In going back over some earlier notes I had made, I saw that I had stated that profiles contain a list of developers who are allowed to sign the profile. This is not entirely true. Generally, each developer will have his or her own certificate–associated with his or her own private key–installed in his/her own OS X Keychain, for use in signing iOS apps. But this is not technically required. An organization could, for example, have a single certificate/private-key combination that every developer in the organization imports into his/her Keychain. This can make life easier, assuming of course that no developer ever leaves the organization.

The sum of the parts

Looking at development profiles as an aggregation of those three parts helps to make sense of why Apple established its requirements. A given iOS app (identified by the App ID), is allowed to be installed on a limited set of devices for testing purposes, and only certain developers identified by their certificates should be allowed to build that app.

What do you need to do?

So how does these concepts translate into practice? Well, let’s assume that you’ve set up your iOS developer account, but aside from that, you’re starting from scratch.
First, you’ll visit https://developer.apple.com and after logging in, make your way to the Identifiers section in your account. Create an App ID for you app. App IDs generally take the reverse-domain-name form, and incorporate the name of the product. For example, yours might look like com.yourcompany.yourproduct  As mentioned above, if you need to use more than one App ID for your product, you can either create multiple App IDs, or else create one using a wildcard (e.g. com.yourcompany.*)
Next, you’ll go to the Certificates section. Follow the instructions to create a new certificate, ensuring at the end that you download the certificate to you Mac. For starters, you’ll want to create a certificate of type iOS App Development. Once finished, double-click the certificate to install it, along with the associated private key, into your Keychain.
Then go to the Devices section, and enter the UUID of any devices on which you’ll want to test your app. There are plenty of resources online, such as this one, to help you find your devices’ UUIDs. Be sure to enter the UUIDs correctly; once entered, you won’t be able to remove any UUIDs for a year.
Finally, you’ll go to the Provisioning Profiles section. Create one of type Development / iOS App Development. When prompted, choose the App ID that you’d just created, select your new certificate, and the UUIDs of any devices on which you want to be able to test.
When finished, be sure to download the profile and double-click it to install it in your copy of Xcode. Alternatively, you can tell Xcode download and install the profile. How to do this seems to change from release to release. To do this in the current release of Xcode 5, you need to open the Preferences window, go to Accounts, enter your account info, then highlight your account name, click View Details, the finally click the refresh button at the bottom-left of the dialog.

When things change

Typically you’ll get this far, and after a bit of stumbling and tweaking things, you’ll get stuff working. But things will change. Here are some common scenarios you’ll need to deal with:

You get a new tester in your organization

This means that you’ll need to add your tester’s UUID to your account. Visit your developer account online, go to the Devices section as you did above, and enter the new UUID(s). Next, you’ll need to revisit the Provisioning Profiles section and edit your profile(s). Add the new devices to the profile. Then either re-download the profile and double-click it to re-import it into Xcode, or just tell Xcode to grab it and re-import it for you (as described above, using the Preferences window).

You get a new developer in your organization

That developer will need to have a certificate/private-key pair installed on their system that can be used to sign your app. As mentioned above, you have two options. The new developer can either create his/her own certificate, and s/he can use an existing one.
For the latter, the developer will have to access your organization’s developer account (you did give them access, right?) and visit the Certificates section to create his/her certificate. S/he will need to download the certificate and install it into his/her Keychain. The provisioning profile(s) will then need to be edited in the Provisioning Profiles section so that the new certificate is added. The developer–and ideally all of the developers in your organization–should then install the certificate into their Xcode instances, either by downloading and double-clicking the certificate, or by refreshing it from within Xcode.

What about Distribution Provisioning Profiles?

Distribution profiles are similar to development profiles, except that they don’t define the devices that are allowed to install the app. This makes sense, since presumably more than 100 pre-defined people will be using your app! Other than that, the concept is the same; your distribution profile must be tied to an app ID, and it must define the certificates that can be used to sign it.

That’s all for now

This article was intended to help shed light on the concepts associated with building iOS apps, rather than providing practical steps. But hopefully it will provide that little bit of of extra insight that will help you keep control over your iOS development process.

What Happens When You Add a Table Cell to a UITableView?

Seasons change. People change. And user interfaces change, usually dynamically.

One dynamic change often seen in iOS apps is the addition or removal of a table cell to a UITableView. Often, you’ll see this when switching a UITableView from read-only to edit mode (for a dramatic example, try editing a contact in the Contacts app.) I’ve done this a few times in some of my iOS apps. I’ve found a few little tutorials out there that describe generally how to do this, but most contain some information gaps. Here, I’ll try to add some of the insight I gained.

I’ll use code examples from my most recent such app. In this app, I have a “Task Details” UITableView in which the user can view the details of a task (think: todo list item). They can also tap an Edit button to edit those details.

Tasks can contain, among other things, a detailed description. Whenever you’re editing a task, a large-ish UITableViewCell containing a UITextView is present, allowing the user to enter or modify a description. However, when in read-only mode, I wanted to remove the UITableViewCell that displays the task’s description if the task in fact has no description (otherwise, it’s wasted space).

The top part of the table view when we’re just looking at the Task. Note that the “Go Shopping” task has no description.

 

Tap the edit button, and the description cell appears. In the real app, of course, it’s animated.

One other important detail for our discussion is that tasks also have simple names. When in read-only mode, of course, I use a standard UITableViewCell (of style UITableViewCellStyleDefault) to display the task’s name. When in edit mode, however, I want to replace that default cell with a custom one containing a UITextField, in which the user can edit the task’s name.

The first thing I did was to create a method called isTaskDescriptionRow:

 

– (BOOL)isTaskDescriptionRow:(int) row {
    if (tableView.editing) {
        return row == TASK_BASICS_DESC_ROW;
    } else {
        if (taskDetailsCellIsShowing) {
            return row == TASK_BASICS_DESC_ROW;
        } else {
            return NO;
        }
    }
}
taskDetailsCellIsShowing is a BOOL that should be self-explanatory. It gets set during the setEditing: animated: call, as shown a little ways below.
Next, I added code like the following to tableView: cellForRowAtIndexPath:
 
…
if (row == TASK_BASICS_NAME_ROW) {
            if (tableView.editing) {
                // create and return a table view cell containing
                // a UITextField for editing the task’s name
            } else {
            // create and return a default table view cell
                // displaying the task’s name
            }

 

} else if ([self isTaskDescriptionRow:row]) {
            UITableViewCell *cell = [[UITableViewCell alloc]
                        initWithStyle: UITableViewCellStyleDefault 
                        reuseIdentifier: nil];
            …
            [descTextView release];
            descTextView = [[UITextView alloc]
                        initWithFrame:CGRectMake(0, 0, w, h)];
            // set descTextView to be non-editable and non-opaque
            descTextView.text = task.description;
            descTextView.userInteractionEnabled = !self.editing;
            [cell.contentView addSubview: descTextView];
            return cell;
} else…
Delegating to isTaskDescriptionRow: makes it easy to determine whether the current NSIndexPath should display the current task’s description. isTaskDescriptionRow: will return YES for the appropriate NSIndexPath if the UITableView is currently in edit mode, or if the current task has a description. If so, then I create a UITextView, configure it, and add it to the current table cell.
As I’d mentioned, when the current task has no description, I want the description row to be added when the table view is entering editing mode, and removed when reverting to read-only mode. This is done in the setEditing: animated: method:
– (void)setEditing:(BOOL)editing animated:(BOOL)animated {
        [super setEditing:editing animated:YES];
        [tableView setEditing:editing animated:YES];
    
        if (editing) {
                if (![self taskHasDescription]) {
                    [self addDescriptionRow];
                } else {
                     [tableView reloadData];
                }
        } else {
               if (![self taskHasDescription]) {
                    [self removeDescriptionRow];
                } else {
                    [tableView reloadData];
                 }
        }
    
        [self.navigationItem setHidesBackButton:editing animated:YES];
        …
}
After invoke setEditing: animated: on the super class as well as the UITableView itself, I check to see if in fact we’re entering editing mode (e.g. if editing = YES). If so, and if the current task does not have a description (i.e. if ![self taskHasDescription]), I invoke my addDescriptionRow method which I will show below. If we’re leaving editing mode, and the current task does not have a description, then I invoke removeDescriptionRow, also shown below:
-(void)addDescriptionRow {
    [tableView beginUpdates];
    taskDetailsCellIsShowing = YES;
    NSIndexPath *idxPath = 
        [NSIndexPath indexPathForRow:TASK_BASICS_DESC_ROW
                        inSection:TASK_BASICS_SECTION];
    NSArray *idxPaths = [NSArray arrayWithObject:idxPath];
    [tableView insertRowsAtIndexPaths:idxPaths
        withRowAnimation:UITableViewRowAnimationFade];
    [tableView endUpdates];
}
 
-(void)removeDescriptionRow {
    [tableView beginUpdates];
    taskDetailsCellIsShowing = NO;
    NSIndexPath *idxPath = 
        [NSIndexPath indexPathForRow:TASK_BASICS_DESC_ROW
                        inSection:TASK_BASICS_SECTION];
    NSArray *idxPaths = [NSArray arrayWithObject:idxPath];
    [tableView deleteRowsAtIndexPaths:idxPaths
        withRowAnimation:UITableViewRowAnimationFade];
    [tableView endUpdates];
}

 

The basic approach in the add and remove methods shown above is to create an array of indexPaths representing the cell(s) to be added or removed. You then pass that array to either insertRowsAtIndexPaths: withRowAnimation: or deleteRowsAtIndexPaths: withRowAnimation:. Finally, Apple’s documentation encourages insertions/deletions to be wrapped within beginUpdates and endUpdates calls, so I do that as well.
That all worked, for the most part; the description cell appeared as desired. But I noticed a bit of a wrinkle. The first tablecell–the one which displays the task’s name–is supposed to transform into an editable UITextField when the Edit button is tapped. But that wasn’t happening.
You can see in the code blocks above that this should be done in tableView: cellForRowAtIndexPath:. When the indexPath represents the task name row (section == TASK_BASICS_SECTION, row == TASK_BASICS_NAME_ROW), I then simply check the value of tableView.editing. If YES, I create a UITextField and add it to the UITableViewCell; if NO, I simply return a default UITableViewCell. Therefore, all I need to do is to add a call to tableView.reloadData in the setEditing: animated: method, right?
Wrong. When I did that, I received the following error:
Terminating app due to uncaught exception ‘NSInternalInconsistencyException’, reason: ‘Invalid update: invalid number of rows in section 0.  The number of rows contained in an existing section after the update (3) must be equal to the number of rows contained in that section before the update (3), plus or minus the number of rows inserted or deleted from that section (1 inserted, 0 deleted).’
Hmm… time to look at my tableView: numberOfRowsInSection: method :
– (NSInteger)tableView:(UITableView *)tv numberOfRowsInSection:(NSInteger)section {
    if (section == TASK_BASICS_SECTION) {
        return [self numTaskDetailsBasicSectionTableRows];
    } else …
    }
}
Which delegates to this method:
– (int)numTaskDetailsBasicSectionTableRows {
    if (tableView.isEditing) {
        return NUM_TASK_DETAILS_BASIC_TABLE_ROWS;
    } else {
        if (taskDetailsCellIsShowing) {
            return NUM_TASK_DETAILS_BASIC_TABLE_ROWS;
        } else {
            return NUM_TASK_DETAILS_BASIC_TABLE_ROWS – 1;
        }
    }
}

 

NUM_TASK_DETAILS_BASIC_TABLE_ROWS represents the number of rows in that top section when the description cell is being shown (i.e. 3). At the point where I call tableView.reloadData, however, it appears that the description cell hasn’t actually been added yet, at least as far as UIKit was concerned.
This led to my primary question: what exactly is UIKit doing when it adds a table view cell? Clearly, it needs to obtain information about the cell being added. This is especially apparent in this case, since the new cell is an atypical, custom cell, both in terms of its content and its height. Yet because the task name cell wasn’t being updated, it didn’t appear to be invoking tableView: cellForRowAtIndexPath: at any point while adding the new cell. So I set a breakpoint at the top of tableView: cellForRowAtIndexPath: to test my theory.
It turns out, UIKit was calling tableView: cellForRowAtIndexPath:. But only once. Specifically, it was calling it for the indexPath I provided in the addDescriptionRow method; i.e. the indexPath of the new description cell.
Pretty clever, actually.
But in my case, it was preventing that task-name cell from being updated. Thus, I still needed to call tableView.reloadData, but I needed to do it after the table cell addition was complete. At this point, I started scouring the documentation, as well as various online discussion forums. What I was looking for was some way to register a callback to be invoked when the table cell was completely added. Unfortunately, I couldn’t find any.
I played around a bit with reordering certain calls and and performing some calls asynchronously. I was able to get the functionality to work, but the animation wound up being far from smooth–quite jarring, in fact. I was also able to get the functionality to work without animation at all, but I really wanted the animation effect. I finally settled on the following:
– (void)setEditing:(BOOL)editing animated:(BOOL)animated {
    … all the other stuff
    if (![self taskHasDescription]) {
        [self performSelector:@selector(reloadNameCell:) 
                   withObject:[NSNumber numberWithInt:0]
                   afterDelay:0.33];
    }
}
 
– (void)reloadNameCell:(NSNumber *)count {
    @try {
        NSIndexPath *ip = [NSIndexPath
                indexPathForRow:TASK_BASICS_NAME_ROW
                        inSection:TASK_BASICS_SECTION];
        [tableView reloadRowsAtIndexPaths:
                [NSArray arrayWithObject:ip]
                withRowAnimation:UITableViewRowAnimationNone];
    }
    @catch (NSException *exception) {
        int i = [count intValue];
        NSLog(@”Exception (%d) in reloading name cell, %@”, 
                        i, [exception description]);
        if (i < 5) {
            [self performSelector:@selector(reloadNameCell:) 
                       withObject:[NSNumber numberWithInt:i + 1]
                       afterDelay:0.125];
        } else {
            NSLog(@”Too many retries; aborting”);
        }
    }
}
If that looks like a hack, that’s because it is. Basically, we wait for 0.33 seconds, which seems to be just enough time for the call-adding animation to complete. We then invoke reloadNameCell: which at its core simply reloads the UITableViewCell that corresponds with the task name. We are also prepared to catch the abovementioned NSInternalInconsistencyException if we didn’t quite wait long enough. If we do catch that exception, we wait again for a short period of time and then try again. We track the number of retries and abort after 5 attempts; at that point, something really went wrong.
So there you have it. Hopefully that provides a little bit of insight as to what goes on under the hood while your UITableView’s structure is smoothly changing.

jQuery Mobile, AJAX, and the DOM

I’ve been messing around with jQuery and jQuery Mobile lately, and ran into an issue that’s worth exploring. It might be a newbie issue–I’m sure seasoned JQM developers don’t fall into this trap much–but I figure that I might be able to save new developers some frustration.

Let me first describe the app I was working on. The app basically provides online utilities that I and other engineers might find useful from time to time in day to day work. The utilities are grouped into different categories (for example, Date Conversion tools, Data Encryption tools, etc). While I started out developing these tools for desktop browsers, I figured I also ought to port them to a mobile web app as well. While the desktop tools are all displayed on one single page, however, the mobile app would present each category of tools (e.g. Date Conversion) on its own page. The “homepage” would present a simple listview, allowing the user to select a category of tools to use.

The homepage is contained in its own HTML page (named, unsurprisingly, index.html). Each tool category also has its own page; for example, the Date Conversion tools are located on dc.html. It was with this page that I ran into problems. The primary date conversion tool is one that takes milliseconds and converts them to a formatted date, and vice-versa. Given this, I’d decided to allow users to choose the date format they want to work with. To implement this, I used a multi-page template in dc.html  The first page div consisted of the tool itself; the second div consisted of a group of radio buttons from which the user could select a date format.

Stripped of irrelevant code, dc.html looks like this:

<!DOCTYPE html>
<html>
<head>
…
function addDfChoices() {
        // This method populates the radio buttons. Its existence becomes important later
}
$(document).ready(function() {
        …
        addDfChoices();
        …
});
…
</head>
<body>


<!– Start of main page –>
<div data-role=”page” id=”main”>
…
  <div data-role=”content” id=”main”>
    <span class=”section_content”>
      <label style=”text-align:baseline”>Millis since 1970</label>
      <a href=”#page_dateformat” style=”width:100px;float:right”>Format</a>
      …
    </span>
  </div><!– /main content –>
</div><!– /main page –>


<div data-role=”page” id=”page_dateformat”>
  …
  <div data-role=”content”>
    <div data-role=”fieldcontain” id=”df_option_container”>
      <fieldset data-role=”controlgroup” id=”df_options”>
      </fieldset>
    </div>
  </div><!– /dateformat content –>
</div><!– /dateformat page –>


</body>
</html>

I loaded dc.html into my browser (I did a lot of testing in Chrome, but also of course on mobile browsers such as mobile Safari on my iPhone). The page worked as expected. On page load, the main page div was displayed, while a small Javascript instantiated different date formats and inserted them into the df_options fieldset located within the hidden page_dateformat page div. Tapping the Format link invoked a slide animation which loaded the radio buttons. Additional Javascript code (not shown above) registered any radio button selections and accordingly modified the date format used in the main page.
Except… sometimes things didn’t work. Specifically, sometimes tapping on the Format link took me to the homepage instead.
Playing around a bit, I learned that if I manually loaded dc.html into my browser, things worked fine. If I started on index.html and then linked to dc.html, and then tapped the Format link, that’s when things went wrong.
This is the relevant code within index.html:
<ul data-role=”listview” data-inset=”true” data-theme=”d”>
  <li><a href=”./dc.html”>Date Conversion</a></li>
  <li><a href=”./de.html”>Data Encoding</a></li>
</ul>
Nothing too special there.
Well, I decided to use Chrome’s DOM inspector to get a look at what was happening. I loaded dc.html and looked in the DOM inspector, and indeed I saw the two major page divs, main and page_dateformat. Then I loaded index.html and tapped on the Date Conversion link, and then took a look again at the DOM inspector after the date conversion tools page loaded. There was the main page div… but where was the page_dateformat div?
Here’s what jQuery was doing. When dc.html itself was loaded, the entire contents of the HTML file were loaded into the DOM. Since I was using a multi-page template, any div declaring data-role=”page” besides the first one (in my case, the page_dateformat div) was hidden… but it was still there in the DOM. The DOM looked roughly like this:
html
  head/
  body
    div data-role=”page” id=”main”/
    div data-role=”page” id=”page_dateformat” display=”hidden”/
  /body
/html
Now, when I started by loading index.html and then tapping the Date Conversion link, dc.html was retrieved but only the first data-role=”page” div was loaded into the DOM. Here, roughly, was the DOM at this point:
html
  head/
  body

    div data-role=”page”/  

            ^– this was the content div for index.html; it’s hidden at this point

    div data-role=”page” id=”main”/  
            ^– this is the first data-role=”page” div in dc.html, loaded via AJAX then inserted into the DOM
  /body
/html
Thus, when I tapped on the Format link (which, if you recall is marked up as <a href=”#page_dateformat”>Format</a>), the browser searched the DOM for a block with an ID of page_dateformat, couldn’t find one, and effectively gave up and reloaded the current page… which at this point was still index.html
At this point I reviewed the jQuery Mobile docs. The following little blurb provides a bit of a clue:

It’s important to note if you are linking from a mobile page that was loaded via Ajax to a page that contains multiple internal pages, you need to add a rel="external" or data-ajax="false" to the link. This tells the framework to do a full page reload to clear out the Ajax hash in the URL. This is critical because Ajax pages use the hash (#) to track the Ajax history, while multiple internal pages use the hash to indicate internal pages so there will be conflicts in the hash between these two modes.

I added data-ajax=”false” to the links on index.html  This solved the problem; dc.html was subsequently loaded completely into the browser’s DOM, allowing the page_dateformat div to be found. However, declaring data-ajax=”false” causes links to lose their animation effects. The pages are simply loaded the old-fashioned way.
I wasn’t too keen about that, so I removed the data-ajax=”false” declarations. I then changed the approach I took to the date format selection. Instead of a new “page” with radio buttons, I simply presented the user with a popup select list. For various reasons, I kept the dfChoices() Javascript function depicted above; the method was now invoked on page load to populate the select with date format options.
And immediately, I encountered the same success rate: when I started on dc.html, things worked great. When I started on index.html and linked to dc.html, the dfChoices() function was never invoked. And of course, the reason was the same as well. Much as the page_dateformat div hadn’t been loaded into the DOM, the dfChoices() function similarly was nowhere to be found.
At this point, I added data-ajax=”false”  back into the homepage’s links; for that matter, I also went back to using the multi-page template with radio buttons. There are certainly other solutions (separating out the radio buttons into their own html page and using some form of local storage to share state? putting Javascript functions into data-role=”page” divs instead of in the page header?) worth exploring, which I’ll do at some point. But just understanding the general issue is a good starting point.